Building a serverless DevOps pipeline for Salesforce with Cloud Build

Last reviewed 2021-02-22 UTC

This tutorial shows you how to build a serverless continuous integration/continuous deployment (CI/CD) pipeline for Salesforce using Salesforce Developer Experience (SFDX) and Cloud Build. Cloud Build pipelines use containerization. The pipelines run builds as a series of build steps, where each build step is run in a Docker container. The pipeline can scale up and down in response to load with no need to pre-provision servers, and it offers fast, consistent, automated builds.

This tutorial is intended for anyone who is responsible for designing, developing, and maintaining DevOps workflows in an organization. These roles can include architects, DevOps teams, and engineers. Different sections of the document illustrate parts of the pipeline for different roles. For example, one part is for administrators and DevOps leads, and another part is for Salesforce developers.

The document assumes that you're familiar with Salesforce DX, the Salesforce CLI, Git, GitHub, Docker, Google Cloud products such as Cloud Build, and containerization concepts. It also assumes that you have a GitHub account.

Software development lifecycles can vary widely; this tutorial assumes that you follow an agile release methodology.

Objectives

  • Set up Salesforce Developer Experience.
  • Set up a Git branching strategy.
  • Configure Cloud Build.
  • Run the CI/CD pipeline for Salesforce using Google Cloud build tools and GitHub.

Costs

This tutorial uses the following billable components of Google Cloud:

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

You might also incur Salesforce costs. In the tutorial, you use a Salesforce Developer Edition org, which might be free. For more information, see the Salesforce page about Developer Edition.

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Google Cloud project.

  3. Enable the Cloud Build API.

    Enable the API

  4. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

  5. Make sure that you have a Salesforce account that can assume the role of the Dev Hub org. If you don't have an org, then you can create a Developer Edition account on the Salesforce developer site.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up

Architecture

The following diagram illustrates the architecture of the CI/CD workflow that you create in this tutorial. In this architecture, projects are organized as releases. Developers who want to work on a feature create a new feature branch from a release branch.

Architecture of the pipeline showing the flow of creating a branch, making a pull request, and merging the change into the main branch. Several steps trigger Cloud Build jobs.

The diagram illustrates the following flow:

  1. Developers create feature branches in GitHub for the features that they are developing.
  2. Developers complete development work and run unit tests in Salesforce scratch orgs.
  3. Developers commit and push their development work to their source code repository (GitHub in this tutorial).
  4. Developers create a pull request to merge their work into the release branch.
  5. The creation of a pull request automatically triggers a Cloud Build job to run tests.
  6. Responsible personnel (usually team leads) review and approve pull requests to merge development work into the release branch.
  7. A merge into the release branch automatically triggers a Cloud Build job to deploy the codebase to QA or to other Salesforce environments.
  8. Optionally, manual tests and reviews are performed in a QA environment.
  9. Responsible personnel create a pull request to merge code into the main branch.
  10. The pull request to the main branch triggers a Cloud Build job to deploy code to production.

Developers who want to work on projects start by cloning the project repository from the enterprise source control tool (GitHub in this tutorial). The following diagram represents this strategy as a graph.

Branching strategy shown as a set of versions, with one branch splitting into multiple feature branches that are then merged separately back into a release branch, and from there into the main branch.

As illustrated in the diagram, the branching strategy consists of the following:

  1. A main branch. The code in the main branch reflects the current version of the code that's running in production.
  2. A release branch. A release branch is a relatively longer-lived branch (compared to a feature branch) that coordinates all changes and code that's pertinent to a release. An organization creates a new release branch for each new release.
  3. One or more feature branches. Feature branches help to isolate work that's in progress from the most up-to-date version of code in the main branch. Typically, several feature branches make up a release branch. It's a good practice for developers to create feature branches for bug fixes.

After developers clone a project repository, they develop on their local machine or in a Salesforce scratch org. They can use a scratch org to run unit tests on the changes they make. If the unit tests pass, they commit their code and push the code into the source code repository. They then generate a pull request for their code to be merged into the parent release branch.

The pull request automatically triggers a Cloud Build job that does the following:

  • Creates a new scratch org to run unit tests.
  • Updates the pull request with the result of the tests.

At this point, team leads and product owners can review the pull request. If the request is approved, the changes are merged into the release branch.

Depending on your software development lifecycle, you can have additional automated steps that are triggered based on a merge to the release branch. Examples of automated steps are deploying the validated code to a higher sandbox, such as quality assurance or system integration testing sandboxes.

You can also configure Cloud Build to send build notifications and perform additional actions using Cloud Functions, Cloud Run, or other Google Cloud tools. (These additional actions are not covered in this tutorial.) This approach provides the flexibility to tailor the pipeline to suit your enterprise DevOps framework.

For simplicity, in this tutorial you deploy the sample codebase to a single Salesforce org (Dev Hub). When you build a CI/CD pipeline for production, you use the architecture illustrated earlier and automate deployments to sandboxes that are part of your software development lifecycle.

Personas who are typically involved in software development

Every organization is different and each has its own array of roles and teams. The following table lists key personas (roles) that typically interact with Salesforce DevOps pipelines like the one described in this tutorial.

People in different roles have different responsibilities for setting up Salesforce pipelines. Therefore, the tutorial has two paths. One path is for administrators and DevOps leads, and the other path is for developers.

Persona Responsibilities
Administrator or DevOps lead
  • Sets up the Dev Hub org.
  • Generates certificates that enable users to connect to the Dev Hub org from the Salesforce CLI.
  • Creates connected apps in all the Salesforce environments to which code is deployed by using the DevOps pipeline.
  • Sets up developer accounts in the Dev Hub org.
  • Sets up the DevOps pipeline and any required triggers.
  • Initializes the source code repository.
  • Sets up Cloud Build triggers.
Salesforce developer
  • Clones the source code repository.
  • Sets up the Salesforce CLI for development.
  • Develops and unit-tests features in a release.
  • When the features are finished, generates a pull request to merge features into the release branch.
QA lead
  • Reviews and approves pull requests to merge a developer's work on a feature into the release branch.
Release lead
  • Manages and approves the pull requests for merging into the main branch, which promotes code to production.

Setting up pipelines for Salesforce administrators and DevOps leads

This section describes the tasks that administrators and DevOps teams follow to set up the CI/CD workflow.

Enable the Dev Hub org

  1. Sign in to your Salesforce production org.
  2. In the Setup tab, in the Quick Find box, enter Dev Hub, and then select Dev Hub.

    The Salesforce Dev Hub page.

  3. Enable Dev Hub.

    This step lets you set up a scratch org. You use the scratch org to deploy the sample code in the tutorial to a Salesforce Developer Edition org.

Create a certificate and key pair

After you enable the Dev Hub org, you need to generate a certificate and key pair that can be used to authenticate to the Salesforce Dev Hub org. You use this certificate and key pair when you configure Cloud Build in later steps.

For production CI/CD pipelines, you need to generate additional certificates to authenticate to Salesforce sandboxes. (You don't create these additional certificates as part of this tutorial.) When you generate certificates and key pairs for each environment, make sure that you give them identifiable names, as in the following examples:

  • Production (Dev Hub org): salesforce.key and salesforce.crt. You use these names in the procedure that follows.
  • Quality Assurance sandbox (QA): salesforce_qa.key and salesforce_qa.crt.
  • Integrated Development sandbox (IDEV): salesforce_dev.key and salesforce_dev.crt.

To generate a certificate and key pair, follow these steps:

  1. In Cloud Shell, generate a certificate and key pair so Cloud Build can authenticate to your Salesforce Dev Hub org from the Salesforce CLI:

    openssl req -x509 -sha256 -nodes -days 36500 -newkey \
    rsa:2048 -keyout salesforce.key -out \
    salesforce.crt
    

    You're prompted to enter details to identify the certificate. For this tutorial, these values aren't important, so press Enter to accept the defaults.

    Notice that in the command, you're using the names salesforce.key and salesforce.crt, because you're creating the certificate and key pair for the Dev Hub org.

  2. Click More and select Download File.

  3. In the Fully qualified file path box, enter the following filename and then click Download:

    salesforce.crt
    

    This step downloads the certificate that you generated to your local machine. You upload the certificate to your Salesforce org in the next section.

Create connected apps in Salesforce

In this section, you create a connected application that Cloud Build can use to deploy your Salesforce code. For this tutorial, you deploy code only to the Dev Hub org. In a production environment, you deploy code for each sandbox and for the production org where you want Cloud Build to deploy your Salesforce code for the DevOps pipeline.

As part of this process, you use the certificate and key pair that you generated in the previous section. The certificate is the public key for authenticating the Salesforce client in a Cloud Shell session to the Salesforce Dev Hub org (Salesforce sandbox). The certificate is also used to authenticate Cloud Build for automated deployments.

The details of some of the steps in the following procedure depend on what edition of Salesforce you're using. Make sure that you use the correct certificate and key pairs for the selected environment.

  1. If you're using Salesforce Lightning Experience, use the App Manager to create connected apps. From Setup in your Salesforce org, do the following:

    1. In the Quick Find box, enter App.
    2. Select App Manager.
    3. Click New Connected App.

    If you're using Salesforce Classic, from Setup in your Salesforce org, do the following:

    1. In the Quick Find box, enter Apps.
    2. Under Build > Create, select Apps.
    3. Under Connected Apps, click New.
  2. For the name of your application, enter Google Cloud DevOps.

    This fills Google_Cloud_DevOps into the API Name box.

  3. Enter the contact email information and any other information that's appropriate for your application.

  4. Select Enable OAuth Settings.

  5. For the Callback URL value, enter the following URL:

    http://localhost:1717/OauthRedirect
    
  6. To enable the option to use digital signatures, click Choose File, and then select the salesforce.crt file that you downloaded earlier.

  7. Add the following OAuth scopes to Selected OAuth Scopes:

    • Access and manage your data (api)
    • Perform requests on your behalf at any time (refresh_token, offline_access)
    • Provide access to your data via the web (web)

    Dialog box selector for selecing OAuth scopes.

  8. Click Save and then click Continue.

  9. Make a note of the Consumer Key value that's displayed in the API section; you need it later when you set up the Cloud Build manifest.

    If you're working in a production environment, you have a key for each deployment environment.

  10. Click Manage, and then to change OAuth policies, click Edit Policies.

  11. Set Permitted users to Admin Approved Users are Pre-Authorized and confirm the choice.

  12. Set IP relaxation to Relax IP Restrictions.

  13. Click Save.

  14. Click Manage Profiles and select the System Administrator option.

    After this step, users who assume this profile can sign in to the Salesforce CLI.

  15. Click Save.

Initialize the Google Cloud environment

  1. In Cloud Shell, set the project that you created or selected as the default project:

    gcloud config set project PROJECT_ID
    

    Replace PROJECT_ID with the ID of your Google Cloud project.

  2. Assign settings for region and zone:

    gcloud config set compute/region us-central1
    gcloud config set compute/zone us-central1-a
    

    In this tutorial, you use us-central1 as the region and us-central1-a as the zone.

  3. Export the current Google project ID to an environment variable named GCP_PROJECT_NUMBER:

    export GCP_PROJECT_NUMBER=$(gcloud projects describe $DEVSHELL_PROJECT_ID --format='value(projectNumber)')
    

Upload Salesforce keys to Cloud Storage

  1. In Cloud Shell, create a Cloud Storage bucket in your build project to store the Salesforce private key files:

    gsutil mb -p ${DEVSHELL_PROJECT_ID} -l us-central1 \
        gs://salesforce-ref-${DEVSHELL_PROJECT_ID}
    

    The bucket must have a globally unique name. This command creates a bucket name that includes your Google Cloud project ID.

  2. Copy the Salesforce private keys that you generated in the Enabling the Dev Hub org section to the new Cloud Storage bucket:

    gsutil cp salesforce.key gs://salesforce-ref-${DEVSHELL_PROJECT_ID}
    

Create your GitHub repository

  1. In Cloud Shell, clone the repository that's associated with this tutorial:

    git clone https://github.com/GoogleCloudPlatform/salesforce-serverless-cicd-cloudbuild
    
  2. Create a new GitHub repository named DEV_REPO_NAME.

    Replace DEV_REPO_NAME with the name you want to assign to the repository locally.

    This is the repository that developers pull code from or push code to. For the purposes of this tutorial, you create this repository in your own GitHub account.

  3. Go to the cloned repository:

    cd salesforce-serverless-cicd-cloudbuild
    
  4. Add your developer GitHub repository as a remote repository:

    git remote add github DEV_REPO_NAME
    

Configure Cloud Build

In this section, you complete the setup steps that are required in order to trigger Cloud Build jobs when developers generate pull requests to merge their work into a release branch.

You set up two triggers for the CI/CD pipeline that you create in this tutorial:

  • A trigger that runs a Cloud Build job when a developer creates a pull request to merge code into the release branch. A typical use for this trigger is to run unit tests.
  • A trigger that runs a Cloud Build job when a pull request is merged into the release branch. A typical use for this trigger is to deploy changes to a destination sandbox (Dev Hub in this tutorial).

Build a base image that includes Salesforce DX

Cloud Build runs your build as a set of steps, where each step runs in a Docker container. You build a base Docker container image that includes the Salesforce CLI, and Cloud Build uses Salesforce CLI commands to run the job.

  1. In Cloud Shell, create a Dockerfile for the image you need to build:

    cat <<EOF > Dockerfile
    FROM debian:buster
    RUN apt-get update && \
    apt-get install -y wget xz-utils
    RUN wget https://developer.salesforce.com/media/salesforce-cli/sfdx-linux-amd64.tar.xz && \
    mkdir sfdx && \
    tar xJf sfdx-linux-amd64.tar.xz -C sfdx --strip-components 1 && \
    ./sfdx/install
    ENTRYPOINT [ "sfdx" ]
    EOF
    
  2. Export the name of the Docker image to an environment variable named SFDX_BASE_IMAGE:

    export SFDX_BASE_IMAGE="gcr.io/${DEVSHELL_PROJECT_ID}/salesforcedx-base-image:1"
    
  3. Build your container with Cloud Build and publish the image to Container Registry:

    gcloud builds submit --tag ${SFDX_BASE_IMAGE}
    

Configure the Cloud Build job

You define a Cloud Build job by editing a cloudbuild.yaml file.

  1. In Cloud Shell, create a cloudbuild.yaml file to define the job steps to run when Cloud Build deploys code to your Salesforce Dev Hub org:

    cat <<EOF > cloudbuild.yaml
    steps:
    - name: gcr.io/cloud-builders/gsutil
      args: ['cp', 'gs://\${_BUCKET_NAME}/salesforce.key', 'salesforce.key']
    - name: "${SFDX_BASE_IMAGE}"
      args:
      - force:auth:jwt:grant
      - --setdefaultusername
      - -u
      - \${_SF_USERNAME}
      - -f
      - ./salesforce.key
      - -i
      - \${_CONSUMER_KEY}
    - name: "${SFDX_BASE_IMAGE}"
      args: ['force:source:deploy', '-p', './force-app/']
    substitutions:
      _BUCKET_NAME: __BUCKET_NAME__
      _SF_USERNAME: __USERNAME__
      _CONSUMER_KEY: __CONSUMER_KEY__
    EOF
    

    The file configures Cloud Build to do the following:

    1. Download the salesforce.key file that Cloud Build uses to authenticate to the Dev Hub org.
    2. Start a Docker container that has the Salesforce CLI installed and then connect to the Dev Hub org by using a JWT grant. Cloud Build uses configuration parameters such as the consumer key and the Salesforce username that's in the Cloud Build trigger definition.
    3. Deploy the code pushed by the developer to the Dev Hub org or to another destination sandbox in production CI/CD pipelines.
  2. Create another file called cloudbuild_pr.yaml to define the job steps to run when Cloud Build deploys code from a pull request to a temporary Salesforce scratch org or sandbox for testing:

    cat <<EOF > cloudbuild_pr.yaml
    steps:
    - name: gcr.io/cloud-builders/gsutil
      args: ['cp', 'gs://\${_BUCKET_NAME}/salesforce.key', 'sales