GitHub and Azure Identities

Recently I had to set up a GitHub Actions workflow for a project that is deployed on Azure, with the codebase maintained on GitHub On Microsoft’s blog post Azure access from GitHub and GitLab pipelines - without secrets. you can find an excellent and comprehensive overview of the authentication method used here. In this post I focus on the practicalities of setting it up and the connection to workload identities on Kubernetes.. I am sure that this is a fairly common development set up, and unlike using Azure DevOps it requires a certain number of manual steps to set up not clearly explained in the official documentation. In this post I will describe the process that I followed to set up resources on Azure and configured the workflow to almost completely automate authentication. The end result feels as seamless as using Azure DevOps.

Goal

First of all, let us specify what we are building. On GitHub we have an organization and a repository. On the Azure side we have a Kubernetes cluster, managed using Azure Kubernetes Service (AKS), and an associated Azure Container Registry (ACR). These are created using azurerm, the Azure provider for Terraform. We would then like to push container images built from a Dockerfile in the GitHub repository to the ACR, which is connected to AKS via a Role Assignment. Furthermore, we would like to enable seamless authentication from GitHub for the Kubernetes Provider for Terraform. Once we have set up the first connection, the second one is straightforward.

These connections use Azure Managed Identities. These managed identities are very useful for our use case, because they do not require management of any resources on Microsoft Entra ID Previously known as Azure Active Directory, or AAD for short. nor they require to create, share or rotate secrets. An additional benefit of this approach is that it only requires the Owner role, rather than the permissions described in the AzureAD Provider for Terraform for azuread_application_registration resources. Specifically we will use a user-assigned managed identity created using azurerm, since these identities are not linked to any one resource. Because these are just like any other identity resource on Azure, their permissions are managed using role assignments.

Note that applications using the Azure libraries, such as those based on the SDK for Python, also benefit from this authentication mechanism. For data analysis, machine learning and similar applications this process is implemented in azure-identity. Packages such as adlfs and fsspec can then use these managed identities when running on AKS in the same way that public GitHub Action runners do.

Azure Container Registry

We begin setting up authentication to the container registry. This exercise will introduce several concepts used throughout and show the practical application of the concepts mentioned above, before moving on to the more complex case involving Kubernetes.

Using Terraform the container registry is created as follows:

resource "azurerm_container_registry" "acr" {
  name                = var.name
  location            = var.location
  resource_group_name = var.resource_group_name
  admin_enabled       = false
}

Note that administrator access is explicitly disabled because all authentication happens as described above. Administrator access relies on username and password credentials, which would go against the goal of not managing secrets ourselves, and may be a security risk if configured improperly or in case of leaked credentials. Let us now create and configure permissions for the managed identity to access this container registry from GitHub. First, the identity resource:

resource "azurerm_user_assigned_identity" "github" {
  name                = "github-identity"
  location            = var.location
  resource_group_name = var.resource_group_name
}

Permissions are granted using role assignments. For pulling and pushing images to a container registry there are two built-in A comprehensive list of built-in role assignments is available on the documentation. role assignments made specifically for this purpose: AcrPull and AcrPush. These are similarly created as Terraform resources:

resource "azurerm_role_assignment" "github_acr_pull" {
  principal_id         = azurerm_user_assigned_identity.github.principal_id
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"

  skip_service_principal_aad_check = true
}

resource "azurerm_role_assignment" "github_acr_push" {
  principal_id         = azurerm_user_assigned_identity.github.principal_id
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPush"

  skip_service_principal_aad_check = true
}

Here skip_service_principal_aad_check = true is set because the identity is created in the same terraform apply. That is precisely the case described in the documentation. Now that we have created an identity and granted it some permissions we can create a set of managed credentials for the GitHub Action to log in to Azure. These are called federated identity credentials. The same Terraform provider is used to create them as before:

resource "azurerm_federated_identity_credential" "github_credentials" {
  name                = "github-actions"
  resource_group_name = var.resource_group_name
  issuer              = "https://token.actions.githubusercontent.com"
  audience            = ["api://AzureADTokenExchange"]
  subject             = var.subject
  parent_id           = azurerm_user_assigned_identity.github.id
}

Here we have introduced several values seemingly out of nowhere. Let us understand what do they mean:

issuer. This is the issuer of the token, in this case GitHub. The issuer is the Identity Provider that verifies that a specific identity is what it claims. Overly simplifying the process, a token This is a JSON Web Token (JWT) a very common way of storing information securely, as they proof that the data they store has not been modified by third parties. signed by GitHub when running an Action is added to the workflow. This token is then provided to Azure as part of the log in step. Then Azure sends this token back to the identity provider to ensure that the identity is trustworthy. After being proved trustworthy Azure allows the client to on resources with the permissions granted above.

audience. In general it is recommended to use the value shown above. For GitHub specifically it must match this value. Additional restrictions on this field and others are described on the workload identity documentation page.

subject. While the two fields above are mostly set by Azure and the external identity provider, the subject should be considered carefully. It indicates to the identity provider what is the scope that this identity can be used More detailed information is available on GitHub documentation regarding subject claims. There several examples are provided both in scope and Cloud providers. Note that specifically on Azure wildcards are not supported, meaning that branches must be individually allowed unless environments or pull requests are used..

To clarify consider these examples:

"repo:<owner>/<name>:ref:refs/heads/main". This subject indicates that Actions run from the main branch of the indicated repository are allowed to log in to Azure. Note that a single branch is referenced like this.

"repo:<owner>/<name>:environment:prod". In this case, any Action that targets the environment with name prod is allowed to log in. GitHub environments allows workflows to be configured with different sets of settings, among other things, as described on the documentation.

"repo:<owner>/<name>:pull-request". Lastly, any Action that runs as part of a pull request is allowed to log in.

Note that these examples are specific to GitHub, other providers require different subjects. In the case of GitLab it is described on this documentation page. It is therefore important to consider what scopes are allowed to use a certain identity, as wrongly configured credentials enables attackers to access critical infrastructure on Azure, or any other Cloud provider.

At last we have created the necessary resources on Azure to allow GitHub Actions to pull and push from the container registry. To summarize, we have created an identity, granted it permissions to the container registry and created a set of credentials that can be used to log in to Azure. The next step is to create secrets on the GitHub repository that indicates to which Azure subscription the Action connects.

We move on now to configure the GitHub Action workflow that will push container images to this container registry. There are three important steps in the workflow job that need to be executed, as well as several secrets that have to defined in the repository and specifying permissions in the job itself.

Job permissions are documented on this page, and we are particularly interested in id-token: write. As described on this other page, granting this permission allows GitHub identity provider to create a JWT. This permission is set in the permissions field, as follows:

name: Push to Azure Container Registry

on:
  pull_request:
    branches: [ main ]

jobs:
  build-and-push:
    name: Build and Push
    runs-on: ubuntu-latest

    permissions:
      id-token: write
      contents: read

    steps: []

Note that this workflow is only executed on pull requests, which means that the subject claim above must be set to "repo:<owner>/<name>:pull-request". Alternatively the deployment environment must be specified in the workflow. It is also possible to grant this permission to the entire workflow. Here we also add contents: read so that actions can access the contents of the repository. Let us proceed to the job steps.

The first step uses the action azure/login@v2 to log in to Azure using the identity credentials created above:

- name: Azure login
  uses: azure/login@v2
  with:
    client-id: ${{ secrets.AZURE_CLIENT_ID }}
    tenant-id: ${{ secrets.AZURE_TENANT_ID }}
    subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

This action executes the equivalent of running az login yourself. The main difference, however, is that it uses the OIDC authentication mechanism described above to log in. Here three secrets are required, the client ID, the tenant ID and the subscription ID. The latter two are obtained and added as secrets to the repository as described here and here respectively. Alternatively, they may be added to an environment or organization as explained in the documentation. The client ID is the unique identifier of github, the managed identity created above.

As an optional verification step, print the account information:

- name: Validate Azure login
  run: |
    az account show

The Terraform Provider for GitHub may be used to create that secret with a resource as follows:

resource "github_actions_secret" "github_client_id" {
  repository      = var.github_repository_name
  secret_name     = "AZURE_CLIENT_ID"
  plaintext_value = azurerm_user_assigned_identity.github.client_id
}

Note that, as described on the provider documentation authentication needs to be set up separately. If these secrets are created in the same repository as the one that builds and pushes the container image then the GITHUB_TOKEN already has the appropriate permissions to do so Depending on the strictness of security practices the permissions of the token may differ..

The second step uses the script action to generate docker credentials for the container registry:

- name: Log in to ACR
  run: |
    az acr login --name ${{ vars.REGISTRY_URL }}

This command internally uses the authentication mechanism we have set up to request temporary credentials specific to the identity to access the container registry In recent versions the Azure CLI also supports generating credentials for tools other than docker such as podman, as described in this section of the documentation.. The registry URL is similarly obtained from the terraform resource that created the container registry and added as repository variable:

resource "github_actions_variable" "registry_url" {
  repository      = var.github_repository_name
  variable_name   = "REGISTRY_URL"
  value           = azurerm_container_registry.acr.login_server
}

The third and last step is to actually build and push the container image, which is achieved with a script:

- name: Build and push
  run: |-
    image_fqdn=${{ vars.REGISTRY_URL }}/$IMAGE_NAME:$IMAGE_TAG

    docker build -t $image_fqdn .
    docker push $image_fqdn

Here $IMAGE_NAME and $IMAGE_TAG are set to the desired values, and it assumes that a Dockerfile is present at the root of the repository. These are details that depend on your specific codebase.

This concludes the procedure to seamlessly connect GitHub Actions and Azure to pull and push container images. Setting up this connection is certainly laborious and requires a certain level of understanding of how this authentication mechanism is used by both Azure and GitHub. The end result is clearly worth it, as no keys need to be stored and rotated, identities are fully managed by Azure, and a high portion of the work is automated with terraform.

Since we have spent this time setting up the connection, let us make more use of it by adding Kubernetes to the mix.

Azure Kubernetes Service

In this section we will not introduce again the same concepts as above, instead we focus now on the connection between GitHub Actions and the Kubernetes cluster so that services can be deployed. First, the Kubernetes cluster as follows:

resource "azurerm_kubernetes_cluster" "kube" {
  name                = var.name
  location            = var.location
  resource_group_name = var.resource_group_name

  oidc_issuer_enabled       = true
  workload_identity_enabled = true

  identity {
    type = "SystemAssigned"
  }
}

Here we have omitted soem fields, such as dns_prefix and kubernetes_version, because they are not relevant. Note that we have enabled both the OIDC issuer and the workload identity extensions by setting oidc_issuer_enabled and workload_identity_enabled to true respectively. These will come back later in the next section. We have additionally set the kubelet identity to be system assigned, because we will assign roles to it.

First we grant permissions to the kubelet identity to pull images from the container registry, as in the previous section using the built-in role AcrPull:

resource "azurerm_role_assignment" "kubelet_acr_pull" {
  principal_id         = azurerm_kubernetes_cluster.kube.kubelet_identity[0].object_id
  scope                = azurerm_container_registry.acr.id
  role_definition_name = "AcrPull"

  skip_service_principal_aad_check = true
}

In this way container images built in GitHub Actions can be pulled and run on Kubernetes without any further configuration. Because we would like to deploy to Kubernetes from GitHub Actions, one possibility is to continue using the Terraform Provider for Kubernetes and the Helm Provider because they manage authenetication themselves.

They are configured as follows:

provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.kube.kube_config.0.host
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.cluster_ca_certificate)
}

provider "helm" {
  kubernetes {
    host                   = data.azurerm_kubernetes_cluster.kube.kube_config.0.host
    client_certificate     = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.client_certificate)
    client_key             = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.kube.kube_config.0.cluster_ca_certificate)
  }
}

Note that this assumes that the Kubernetes cluster is created using a different plan, as recommended . This terraform plan can be applied from GitHub Actions, with authentication done as described in the previous section by setting these environment variables in the workflow:

env:
  ARM_CLIENT_ID: ${{ secrets.ARM_CLIENT_ID }}
  ARM_SUBSCRIPTION_ID: ${{ secrets.ARM_SUBSCRIPTION_ID }}
  ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
  ARM_USE_AZUREAD: true
  ARM_USE_OIDC: true

Note the different prefix, with the ARM_ prefix being used rather than AZURE_. Setting ARM_USE_AZUREAD and ARM_USE_OIDC to true, indicates to the terraform provider that it can use the managed credentials configured above. The ARM_CLIENT_ID, ARM_SUBSCRIPTION_ID and ARM_TENANT_ID are obtained in the same way as described in the previous section, with the addition that the associated identity must have the Azure Kubernetes Service Cluster Admin Role role asignment to be allowed to generate the necessary cluster credentials It would be possible to use role-based access control (RBAC) on Kubernetes as described in the identity concepts page. See also how to manage access.. It is good advice to create separate identities for pulling and pushing to the container registry and to deploy to Kubernetes, as the latter is much more security critical.

Then we can deploy an application using Helm from Terraform. For example:

resource "helm_release" "argocd" {
  name       = "argocd"
  namespace  = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"

  create_namespace = true
}

This will create the namespace argocd and deploy ArgoCD using the Helm chart, as an example application Did you know that ArgoCD can manage itself? However I recommend using the Helm chart deployed here with terraform rather than the method described in their documentation..

With this setup we can now build and deploy applications to the Kubernetes cluster from GitHub Actions. However, we are not done yet. There is one more use case that we would like to cover, and that is to enable seamless authentication for the applications themselves deployed on Kubernetes. The next section covers this last aspect.

Microsoft Entra Workload ID

In this last section we will apply everything we have seen previously to address a permissions problem that commonly arises on Kubernetes workloads This section closely follows this page on the documentation. The next pages are also used.. Unlike services running on Azure resources, such as a virtual machine hosting an application, it is not possible in the same manner to associate the identity of the Azure resource to the application. On Kubernetes, however, a single VM runs multiple workloads with very likely permissions most of the time. One way to provide these workloads with the needed permissions is to create an identity on AzureAD with the associated credentials and inject those into the environment. This approach suffers from the issues listed at the start of this text.

An alternative approach is to use the same authentication mechanism as for GitHub Actions. Conceptually both situations are almost the same, with an identity provider that asserts the identity of the workload running in an untrusted environment. In the previous sections that is the GitHub Action runner while in the current section is the workload running on Kubernetes. While the identity provider above is GitHub, as specified by the issuer field, for Kubernetes we will use the OIDC provider managed by Azure that we have enabled by setting oidc_issuer_enabled = true on terraform.

With this information we can then create a new managed identity with federated credentials and a role assignment. As a practical example, let us grant permissions to access a storage account to Kubernetes pods created from a hybrid Dagster+ deployment. This is a fairly common scenario for data ingestion or other transformation pipelines that read or write data in a storage account See also this page on the documentation for an example that grants access to a Key Vault.. Using terraform, these resources are:

resource "azurerm_user_assigned_identity" "dagster" {
  name                = "dagster-identity"
  location            = var.location
  resource_group_name = var.resource_group_name
}

resource "azurerm_federated_identity_credential" "dagster_aks" {
  name                = "kubernetes-dagster-plus"
  resource_group_name = var.resource_group_name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.kube.oidc_issuer_url
  subject             = "system:serviceaccount:dagster-plus:dagster"
  parent_id           = azurerm_user_assigned_identity.dagster.id
}

resource "azurerm_role_assignment" "dagster_storage_blob_data_contributor" {
  scope                = azurerm_storage_account.data.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.dagster.principal_id
}

This will create a new identity, a federated credential and a role assignment that grants permissions to read and write blobs in the storage account. Note that we assume that the resource azurerm_storage_account.data has already been created on Azure. Also note that the issuer field refers to the managed OIDC managed identity provider that is specific to this Kubernetes cluster. Field subject in resource azurerm_federated_identity_credential is now different from that of GitHub, since it now refers to a Kubernetes service account, not the scope of a GitHub Action. This subject claim refers to a service account with name dagster in namespace dagster-plus. If the service account does not exist it can be created by applying a terraform plan from a GitHub Actions workflow set up as described in the previous section. The terraform resource is:

resource "kubernetes_namespace" "dagster_plus" {
  metadata {
    name = "dagster-plus"
  }
}

resource "kubernetes_service_account" "dagster" {
  metadata {
    name      = "dagster"
    namespace = kubernetes_namespace.dagster_plus.metadata[0].name
    annotations = {
      "azure.workload.identity/client-id" = azurerm_user_assigned_identity.dagster.client_id
    }
  }
}

The annotation azure.workload.identity/client-id is the unique identifier of the managed identity created above. It is set as described on the documentation and is used by the mutating webhook that is installed by Azure on the Kubernetes cluster when field workload_identity_enabled is set to true If you would like to check it out yourself, get the webhook resource definition with command kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io azure-wi-webhook-mutating-webhook-configuration -o yaml.. With this configuration, pods with this service account can authenticate to Azure and act with the permissions granted to the identity created above. The Kubernetes documentation provides more information. As explicitly stated in the documentation for workload identity, the mutating webhook relies on adding a specific label to pods.

Specifically for Dagster+, the code location can be configured as explained on the Dagster documentation:

location_name: LOCATION_NAME
image: IMAGE
code_source:
  module_name: MODULE_NAME
container_context:
  k8s:
    env_vars:
    - AZURE_STORAGE_ANON=false
    - AZURE_STORAGE_ACCOUNT_NAME=STORAGE_ACCOUNT_NAME
    run_k8s_config:
      pod_spec_config:
        serviceAccountName: dagster
      pod_template_spec_metadata:
        labels:
          azure.workload.identity/use: "true"

In this way runs launched from Dagster are executed as Kubernetes pods with the correct service account and label. The two environment variables set above are for adlfs as explained here. It is necessary to set AZURE_STORAGE_ANON=false so that anonymous authentication is not attempted, as it will fail, and the name of the storage account must be provided either as environment variable or in the Python code itself. Note how in the pod spec we set both the label and the service account as described above.

Then, the storage account is accessed as:

import fsspec

fs = fsspec.filesystem(protocol="az", account_name="STORAGE_ACCOUNT_NAME")

print(fs.ls("/"))

Executing this code lists the contents of the root directory of the storage account after authenticating using the managed identity, federated credentials and OIDC explained in this post.

Conclusion

In this post we have explained in detail a method for seamless and secure authentication in GitHub Actions and Kubernetes workloads. As a result, we avoid manually handling credentials and secrets. Fundamentally we offload the burden of asserting trust to a managed identity provider rather than relying on a secret provided by the application. Scaling up this solution is also much simpler than the alternative, as managed identities, federated credentials and role assignments are defined as code, allowing for a large degree of automation.

In general Python applications running on Kubernetes can benefit from this solution significantly, as they likely already depend on libraries that integrate with Azure authentication. In this post we have mentioned Dagster, which can be used for easily creating machine learning and data transformation pipelines. It is to be expected that in both cases access to storage accounts is needed, which can be managed as described in this post.