Host GitHub runners with Cloud Run worker pools

Retrieve the code sample

To retrieve the code sample for use:

1. Clone the sample repository to your local machine:

   git clone https://github.com/GoogleCloudPlatform/cloud-run-samples
   

2. Change to the directory that contains the Cloud Run sample code:

   cd cloud-run-samples/github-runner
   

Understand the core code

The sample is implemented as a worker pool, and autoscaler, described next.

Worker pool

The worker pool is configured with a Dockerfile that is based on the GitHub-created actions/runner image.

All the logic is self-contained in this image, apart from a small helper script.

FROM ghcr.io/actions/actions-runner:2.328.0

# Add scripts with right permissions.
USER root
# hadolint ignore=DL3045
COPY start.sh start.sh
RUN chmod +x start.sh

# Add start entrypoint with right permissions.
USER runner
ENTRYPOINT ["./start.sh"]

This helper script runs when the container is started, registering itself to the configured repository as an ephemeral instance, using a token you will create. The script also defines what actions to take when the container is scaled down.

# Configure the current runner instance with URL, token and name.
mkdir /home/docker/actions-runner && cd /home/docker/actions-runner
echo "GitHub Repo: ${GITHUB_REPO_URL} for ${RUNNER_PREFIX}-${RUNNER_SUFFIX}"
./config.sh --unattended --url ${GITHUB_REPO_URL} --pat ${GH_TOKEN} --name ${RUNNER_NAME}

# Function to cleanup and remove runner from Github.
cleanup() {
   echo "Removing runner..."
   ./config.sh remove --unattended --pat ${GH_TOKEN}
}

# Trap signals.
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

# Run the runner.
./run.sh & wait $!

Autoscaler

The autoscaler is a function that scales up the worker pool when there is a new job in the queue, or scales down when a job is complete. It uses the Cloud Run API to check the current number of workers in the pool, and adjusts that value as required.

try:
    current_instance_count = get_current_worker_pool_instance_count()
except ValueError as e:
    return f"Could not retrieve instance count: {e}", 500

# Scale Up: If a job is queued and we have available capacity
if action == "queued" and job_status == "queued":
    print(f"Job '{job_name}' is queued.")

    if current_instance_count < MAX_RUNNERS:
        new_instance_count = current_instance_count + 1
        try:
            update_runner_instance_count(new_instance_count)
            print(f"Successfully scaled up to {new_instance_count} instances.")
        except ValueError as e:
            return f"Error scaling up instances: {e}", 500
    else:
        print(f"Max runners ({MAX_RUNNERS}) reached.")

# Scale Down: If a job is completed, check to see if there are any more pending
# or in progress jobs and scale accordingly.
elif action == "completed" and job_status == "completed":
    print(f"Job '{job_name}' completed.")

    current_queued_actions, current_running_actions = get_current_actions()
    current_actions = current_queued_actions + current_running_actions

    if current_queued_actions >= 1:
        print(
            f"GitHub says {current_queued_actions} are still pending."
            f"Won't change scaling ({current_instance_count})."
        )
    elif current_queued_actions == 0 and current_running_actions >= 1:
        print(
            f"GitHub says no queued actions, but {current_running_actions} running actions."
            f"Won't change scaling ({current_instance_count})."
        )
    elif current_actions == 0:
        print(f"GitHub says no pending actions. Scaling to zero.")
        update_runner_instance_count(0)
        print(f"Successfully scaled down to zero.")
    else:
        print(
            f"Detected an unhandled state: {current_queued_actions=}, {current_running_actions=}"
        )
else:
    print(
        f"Workflow job event for '{job_name}' with action '{action}' and "
        f"status '{job_status}' did not trigger a scaling action."
    )

Configure IAM

This tutorial uses a custom service account with the minimum permissions required to use the provisioned resources. To set up the service account, do the following:

1. Set your project ID in gcloud:

   gcloud config set project PROJECT_ID
   

   Replace PROJECT_ID with your project ID.

2. Create a new Identity and Access Management service account:

   gcloud iam service-accounts create gh-runners
   

3. Grant the service account permissions to act as a service account on your project:

   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member "serviceAccount:gh-runners@PROJECT_ID.iam.gserviceaccount.com" \
     --role=roles/iam.serviceAccountUser
   

   Replace PROJECT_ID with your project ID.

Retrieve GitHub information

The GitHub documentation for adding self-hosted runners suggests adding runners through the GitHub website, which then provides a specific token to use for authentication.

This tutorial will dynamically add and remove runners, and needs a static GitHub token to do so.

To complete this tutorial, you need to create a GitHub token with access to interact with your selected repository.

Identify the GitHub repository

In this tutorial, the GITHUB_REPO variable represents the repository name. This is the part of the GitHub repository name that after the domain name, for both personal user repositories and organization repositories.

You will be referencing the repository name that comes after the domain name for both user-owned and organization-owned repositories.

In this tutorial:

• For https://github.com/myuser/myrepo, the GITHUB_REPO is myuser/myrepo.
• For https://github.com/mycompany/ourrepo, the GITHUB_REPO is mycompany/ourrepo.

Create access token

You need to create an access token on GitHub, and securely save it in Secret Manager:

1. Ensure you are logged into your GitHub account.
2. Navigate to GitHub's Settings > Developer Settings > Personal Access Tokens page.
3. Click Generate new token, and select Generate new token (classic).
4. Create a new token with the "repo" scope.
5. Click Generate token.
6. Copy the generated token.

Create secret value

Take the secret token you just created, and store it in Secret Manager, and set access permissions.

1. Create the secret in Secret Manager:

   echo -n "GITHUB_TOKEN" | gcloud secrets create github_runner_token --data-file=-
   

   Replace the GITHUB_TOKEN with the value you copied from GitHub.

2. Grant access to your newly created secret:

   gcloud secrets add-iam-policy-binding github_runner_token \
     --member "serviceAccount:gh-runners@PROJECT_ID.iam.gserviceaccount.com" \
     --role "roles/secretmanager.secretAccessor"
   

Deploy Worker Pool

Create a Cloud Run worker pool to process GitHub actions. This pool will use an image based on the GitHub-created actions/runner image.

Set up Cloud Run worker pool

1. Navigate to the sample code for the worker pool:

   cd worker-pool-container
   

2. Deploy the worker pool:

   gcloud beta run worker-pools deploy WORKER_POOL_NAME \
     --region WORKER_POOL_LOCATION \
     --source . \
     --scaling 1 \
     --set-env-vars GITHUB_REPO=GITHUB_REPO \
     --set-secrets GITHUB_TOKEN=github_runner_token:latest \
     --service-account gh-runners@PROJECT_ID.iam.gserviceaccount.com \
     --memory 2Gi \
     --cpu 4
   

   Replace the following:

   • WORKER_POOL_NAME the name of the worker pool
   • WORKER_POOL_LOCATION the region of the worker pool
   • GITHUB_REPO the identified GitHub repo name
   • PROJECT_ID the Google Cloud project ID

   If this is the first time using Cloud Run source deploys in this project, you will be prompted to create default Artifact Registry repository.

Using worker pool

You now have a single instance in your worker pool, ready to accept jobs from GitHub actions.

To verify the you have completed the setup of your self-hosted runner, invoke a GitHub action on your repository.

For your action to use your self-hosted runners, you need to change a GitHub action's job. In the job, change the runs-on value to self-hosted.

If your repo doesn't already have any actions, you can follow the Quickstart for GitHub Actions.

Once you can configured an action to use the self-hosted runners, run the action.

Confirm the action completes successfully in the GitHub interface.

Deploy GitHub Runner Autoscaler

You deployed one worker in your original pool, which will allow processing of one action at a time. Depending on your CI usage, you may need to scale your pool to handle an influx of work to be done.

Once you deploy the worker pool with an active GitHub runner, configure the autoscaler to provision worker instances based on the job status in the actions queue.

This implementation listens for a workflow_job event. When workflow job is created, it will scale up the worker pool, and once the job is completed, scale it down again. It won't scale the pool beyond the maximum number of instances configured, and will scale to zero when all running jobs have completed.

You can adapt this autoscaler based on your workloads.

Create webhook secret value

To create a secret value for the webhook, do the following:

1. Create a Secret Manager secret containing an arbitrary string value.

   echo -n "WEBHOOK_SECRET" | gcloud secrets create github_webhook_secret --data-file=-
   

   Replace WEBHOOK_SECRET with an arbitrary string value.

2. Grant access to the secret to the autoscaler service account:

   gcloud secrets add-iam-policy-binding github_webhook_secret \
     --member "serviceAccount:gh-runners@PROJECT_ID.iam.gserviceaccount.com" \
     --role "roles/secretmanager.secretAccessor"
   

Deploy the function to receive webhook requests

To deploy the function for receiving webhook requests, do the following:

1. Navigate to the sample code for the webhook:

   cd ../autoscaler
   

2. Deploy the Cloud Run function:

   gcloud run deploy github-runner-autoscaler \
     --function github_webhook_handler \
     --region WORKER_POOL_LOCATION \
     --source . \
     --set-env-vars GITHUB_REPO=GITHUB_REPO \
     --set-env-vars WORKER_POOL_NAME=WORKER_POOL_NAME \
     --set-env-vars WORKER_POOL_LOCATION=WORKER_POOL_LOCATION \
     --set-env-vars MAX_RUNNERS=5 \
     --set-secrets GITHUB_TOKEN=github_runner_token:latest \
     --set-secrets WEBHOOK_SECRET=github_webhook_secret:latest \
     --service-account gh-runners@PROJECT_ID.iam.gserviceaccount.com \
     --allow-unauthenticated
   

   Replace the following:

   • GITHUB_REPO the part of your GitHub repository name after the domain name
   • WORKER_POOL_NAME the name of the worker pool
   • WORKER_POOL_LOCATION the region of the worker pool
   • REPOSITORY_NAME the GitHub repository name

3. Note the URL your service was deployed to. You will use this value in a later step.

4. Grant the service account permissions to update your worker pool:

   gcloud alpha run worker-pools add-iam-policy-binding WORKER_POOL_NAME \
     --member "serviceAccount:gh-runners@PROJECT_ID.iam.gserviceaccount.com" \
     --role=roles/run.developer
   

   Replace PROJECT_ID with your project ID.

Create GitHub webhook

To create the GitHub webhook, follow these steps:

1. Ensure you are logged into your GitHub account.
2. Navigate to your GitHub repository.
3. Click Settings.
4. Under "Code and automation", click Webhooks.
5. Click Add webhook.

6. Enter the following:

   1. In Payload URL, enter the URL of the Cloud Run function you deployed earlier.

      The URL will look like: https://github-runner-autoscaler-PROJECTNUM.REGION.run.app, where PROJECTNUM is the unique numeric identifier of your project, and REGION is the region you deployed the service to.

   2. For Content type, select application/json.

   3. For Secret, enter the WEBHOOK_SECRET value you created previously.

   4. For SSL verification, select Enable SSL verification.

   5. For "Which events would you like to trigger this webhook?", select Let me select individual events.

   6. In the event selection, select Workflow jobs. Unselect any other option.

   7. Click Add webhook.

Scale down your worker pool

The webhook is now in place, so you don't have to have a persistent worker in the pool. This will also ensure you have no running workers when there is no work to be done, reducing costs.

• Adjust your pool to scale to zero:

  gcloud beta run worker-pools update WORKER_POOL_NAME \
    --region WORKER_POOL_LOCATION \
    --scaling 0
  

Use your autoscaling runner

To verify your autoscaling runner is working correctly, run an action you previously configured to runs-on: self-hosted.

You can track progress of your GitHub Actions on the "Actions" tab of your repository.

You can check the execution of your webhook function and worker pool by checking the Logs tab of the Cloud Run function and Cloud Run worker pool respectively.