Using Recommendations for Infrastructure as Code

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See Cleaning up for more detail.

Set up your environment

1. In Google Cloud console, select your build project.

2. In the Google Cloud console, go to Cloud Shell.

   Go to Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed, and with values already set for your current project. It can take a few seconds for the session to initialize.

   Use Cloud Shell for all of the terminal commands in this tutorial.

3. Create an environment variable to hold the project number for your build project using the command below:

   export BUILD_PROJECT_ID=$DEVSHELL_PROJECT_ID
   

4. Create an environment variable to hold the project number for your test project . Copy the test project ID manually and replace PROJECT-ID with it,

   export TEST_PROJECT_ID=PROJECT-ID

5. You assign default settings for values that are used throughout the tutorial, such as region and zone. In this tutorial, you use us-central1 as the default region and us-central1-b as the default zone.

6. Set the default region and zone for this tutorial by running the following command:.

   gcloud config set compute/zone us-central1-b --project $BUILD_PROJECT_ID
   gcloud config set compute/zone us-central1-b --project $TEST_PROJECT_ID
   

7. Set your build project as the default project:

   gcloud config set project $BUILD_PROJECT_ID
   

8. Create an environment variable calledBUILD_PROJECT_NUMBER for yourbuildproject number

   export BUILD_PROJECT_NUMBER=$(gcloud projects describe $DEVSHELL_PROJECT_ID --format='value(projectNumber)')
   

9. Clone the GitHub repository for this tutorial:

Create Bucket for Terraform State

Create a Cloud Storage bucket in your build project to store the Terraform state file.

gcloud storage buckets create gs://recommender-tf-state-$BUILD_PROJECT_ID \
  --project=${BUILD_PROJECT_ID} --location=us-central1

Create a GitHub repository

Create a GitHub repository to serve as the sample IaC repository

1. Create a new private GitHub repository. This IAC-REPO-NAME repository serves as your IaC repository for the purposes of this tutorial

2. In the following setps, you will push the files in the sample-iac sub-directory of the cloned repository to your GitHub account.

   1. In Cloud Shell, copy the sample-iac directory to your home directory. You will use this directory to create a new local repository and push that to GitHub.

      cp -r recommender-iac-pipeline-nodejs-tutorial/sample-iac $HOME
      

   2. Navigate to the new directory

      cd $HOME/sample-iac
      

   3. Initialize the repository in your local machine.

      git init
      

   4. Add IAC-REPO-NAME as the remote repository, replace IAC-REPO-NAME and GITHUB-ACCOUNT with appropriate values

      git remote add origin https://github.com/GITHUB-ACCOUNT/IAC-REPO-NAME

   5. Replace the placeholders in the files in this repository with yourtestproject ID and the Terraform Cloud Storage bucket name.

      sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./terraform.tfvars
      
      sed -i "s|__STATE_BUCKET_NAME__|recommender-tf-state-$BUILD_PROJECT_ID|g" ./backend.tf
      

   6. Add, commit, and push to GitHub.

      git add .
      git commit -m "First Commit"
      git push origin master
      

   7. Sign in to your GitHub account when prompted.

Generate SSH keys for your repository

Set up SSH key authentication with your IaC repository in GitHub and upload the keys to Cloud Storage.

1. Generate SSH keys for your GitHub repository.

   1. Generate an SSH key pair. Replace your_email@example.com with your GitHub email address. In Cloud Shell:

      ssh-keygen -t rsa -b 4096 -m PEM -C "your_email@example.com"

   2. When you're prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.

   3. At the prompt to enter a passphrase, press Enter.

2. Make note of the directory SSH-KEYS-DIR that you save the downloaded SSH keys in. By default the location is$HOME/.ssh/

3. Copy the SSH public key that you generated to your GitHub repository as a Deploy Key.

   1. Copy the SSH public key that you generated in Cloud Shell. Replace SSH-KEYS-DIR with your directory path.

      cat SSH-KEYS-DIR/id_rsa.pub

   2. In your GitHub account, navigate to the IAC-REPO-NAME repository

   3. Click Settings > Deploy Keys.

   4. Click Add Deploy Key and paste in the SSH public key you copied. Choose a Title for the key.

   5. Select the check box "Allow write access"

   6. Click Save.

4. Navigate back to your Cloud Shell session

5. Create theknown_hosts file for GitHub. In your Cloud Shell session, run the command:

   ssh-keyscan github.com >> ~/.ssh/known_hosts
   

6. Create a Cloud Storage bucket in yourbuild project and upload your SSH keys and known_hosts file to it. Replace SSH-KEYS-DIR with the path to the directory where you generated the SSH keys.

   gcloud storage buckets create gs://github-keys-$BUILD_PROJECT_ID --project=${BUILD_PROJECT_ID} --location=us-central1
   
   gcloud storage cp SSH-KEYS-DIR/id_rsa* gs://github-keys-$BUILD_PROJECT_ID
   gcloud storage cp SSH-KEYS-DIR/known_hosts gs://github-keys-$BUILD_PROJECT_ID

7. Generate a Personal Access Token for GitHub. This token is used when performing Git operations using API calls that the recommender-parser service makes to generate pull requests, check-in updated IaC manifests.

   1. In your GitHub account, in the upper-right corner of any page, click your profile photo, then click Settings.

   2. In the left sidebar, click Developer settings.

   3. In the left sidebar, click Personal access tokens

   4. Click Generate new token.

   5. Give your token a descriptive name.

   6. Select the scopes as repo.

   7. Click Generate token.

   8. Copy the token to your clipboard.

   9. In your Cloud Shell session, create an environment variable.

      export GITHUB_PAT=personal-access-token-you-copied

Set up Cloud Build

1. Connect your IAC-REPO-NAME Git repository to integrate with Cloud Build.

   1. Go to the Cloud Build App page in the GitHub Marketplace.
   2. Scroll down and click Setup with Google Cloud Build at the bottom of the page.
   3. If prompted, Sign in to GitHub.
   4. Select Only select repositories. Use the Select repositories drop-down list to only enable access to your IAC-REPO-NAME in the Cloud Build app.
   5. Click Install.

   6. Sign in to Google Cloud.

      The Authorization page is displayed where you are asked to authorize the Google Cloud Build app to connect to Google Cloud.

   7. Click Authorize Google Cloud Build by GoogleCloudBuild. You are redirected to the Google Cloud console.

   8. Select your Google Cloud project.

   9. Enable the consent checkbox and click Next.

   10. In the Select repository page that appears, select the IAC-REPO-NAME GitHub repo

   11. Click Connect repository.

   12. Click Create a Trigger. This creates a trigger definition for you.

   13. Click Create to save your build trigger.

   For more information, see Running builds on GitHub.

2. The directory that you copied has a cloudbuild.yaml file. This configuration file outlines the steps that a Cloud Build job executes when triggered.

   steps:
   - name: hashicorp/terraform:0.12.0
     args: ['init']
   - name: hashicorp/terraform:0.12.0
     args: ['apply', '-auto-approve']
   

3. Add permissions to your Cloud Build service account to allow it to create service accounts, associate roles, and virtual machines (Compute Engine instances) in the test project

   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:$BUILD_PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
     --role roles/compute.admin \
     --project $TEST_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:$BUILD_PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
     --role roles/iam.serviceAccountAdmin \
     --project $TEST_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
   --member serviceAccount:$BUILD_PROJECT_NUMBER@cloudbuild.gserviceaccount.com \
   --role roles/iam.securityAdmin \
   --project $TEST_PROJECT_ID
   

4. Open the Build Triggers page in the Google Cloud console.

5. Select build project, click Open.

6. Update the trigger's definition:

   1. Click more_vert menu and then click Edit.
   2. For Configuration, select the Cloud Build configuration file (yaml or json) option and type in cloudbuild.yaml in the text field.
   3. Click Save.

7. To manually test the build trigger, click Run on your trigger's entry in the triggers list.

8. Verify that a Compute Engine instance called tf-compute-1 and a service account called Terraform Recommender Test are created in your test project by the Cloud Build job you ran in the previous step

Deploy the recommender-parser Cloud Run service

1. In Cloud Shell, Change directories to the directory created by cloning the repository

   cd $HOME/recommender-iac-pipeline-nodejs-tutorial/parser-service
   

2. Configure Google Cloud to use a default region for Cloud Run services. In this tutorial, you use the us-central1 region but you can choose a different supported region if you prefer.

   gcloud config set run/region us-central1
   

3. The parser-servicedirectory has a stub sub-directory which has a few sample payload JSONs for you to test the recommender-parser service with. Run the following sed commands to replace the PROJECT_ID placeholders in these JSONs with your test project ID.

   sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./stub/iam.json
   sed -i "s|__PROJECT_ID__|${TEST_PROJECT_ID}|g" ./stub/vm.json
   

4. Run the following command to create an environment variable for your Docker image.

   export IMAGE=gcr.io/$BUILD_PROJECT_ID/recommender-parser:1.0
   

5. Build the image and upload to Container Registry

   gcloud builds submit --tag $IMAGE .
   

6. Create a service account for the recommender-parser service to interact with other Google Cloud services in the pipeline. It is a good practice to grant granular permissions to your Cloud Run services, refer to Cloud Run service identity for more details.

   gcloud beta iam service-accounts create recommender-parser-sa \
     --description "Service account that the recommender-parser service uses to invoke other Google Cloud services" \
     --display-name "recommender-parser-sa" \
     --project $BUILD_PROJECT_ID
   

7. The recommender-parser service needs to access the GitHub SSH keys and Terraform state you uploaded to Cloud Storage buckets created earlier. Add the service account as a member to the Cloud Storage bucket.

   gcloud storage buckets add-iam-policy-binding gs://github-keys-$BUILD_PROJECT_ID \
     --member=serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/storage.objectUser
   
   gcloud storage buckets add-iam-policy-binding gs://recommender-tf-state-$BUILD_PROJECT_ID \
     --member=serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/storage.objectUser
   

8. Give the recommender-parser service's service account access to Firestore, Recommender, and the Service Usage API.

   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/datastore.user
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.iamAdmin
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.iamViewer
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/recommender.computeAdmin
   
   gcloud projects add-iam-policy-binding $TEST_PROJECT_ID \
     --member serviceAccount:recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/serviceusage.serviceUsageConsumer
   

9. Deploy the Cloud Run service, which is called recommender-parser, by running the command. Replace GITHUB-ACCOUNT with your GitHub account username, not email. Accept any system prompts.

   gcloud run deploy \
    --image=${IMAGE} \
    --no-allow-unauthenticated \
    --region us-central1 \
    --platform managed \
    --service-account recommender-parser-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
    --set-env-vars="GITHUB_ACCOUNT=github.com:GITHUB-ACCOUNT,GITHUB_PAT=${GITHUB_PAT},SSH_KEYS_BUCKET=github-keys-${BUILD_PROJECT_ID},TERRAFORM_STATE_BUCKET=recommender-tf-state-$BUILD_PROJECT_ID" \
    --project $BUILD_PROJECT_ID \
    recommender-parser

Set up Firestore

1. In Google Cloud console, in your build project, navigate to the Firestore page.
2. When prompted for mode selection, click Select Native Mode.
3. Selectus-east1 as the default location.
4. Click Create Database.

Therecommender-parser service writes documents to this database for the following purposes:

• To keep track of the recommendations it has retrieved from the Recommender API
• Call the Recommender API once the recommendations are processed to update the status of each processed recommendation to SUCCEEDED or FAILED as appropriate. This is a key step that makes the pipeline idempotent by ensuring that recommendations are not processed incompletely or multiple times.

Set up a Cloud Scheduler job

1. Create a service account that Cloud Scheduler jobs use to run the recommender-parser service.

   gcloud beta iam service-accounts create recommender-scheduler-sa \
     --description "Service Account used by Cloud Scheduler to invoke the recommender-parser service" \
     --display-name "recommender-scheduler-sa" \
     --project $BUILD_PROJECT_ID
   

2. Give the service account run/invoker role to be able to invoke the Cloud Run service.

   gcloud beta run services add-iam-policy-binding recommender-parser \
   --member=serviceAccount:recommender-scheduler-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
   --role=roles/run.invoker \
   --region=us-central1
   

3. Get your recommender-service URL:

   gcloud beta run services list --platform managed --project $BUILD_PROJECT_ID
   

   Your Cloud Scheduler job invokes the recommender-parser service's /recommendation/iam route to parse IAM recommendations and the /recommender/vm route to parse VM sizing recommendations.

4. Create a variable for the endpoint that Cloud Scheduler jobs invoke. Replace RECOMMENDER-SERVICE-URL with the recommender-service URL you copied in the previous step.

   export RECOMMENDER_ROUTE_TO_INVOKE_IAM=RECOMMENDER-SERVICE-URL/recommendation/iam

   You URL should look like this sample URL after appending the route information:

   RECOMMENDER-SERVICE-URL/recommendation/iam
   

5. Create a Cloud Scheduler job called recommender-iam-scheduler.

   • Change the selected time-zone based on your location.
   • Replace IAC-REPO-NAME with the name of the GitHub repository you created.

   The message body takes three inputs and you must construct it as outlined below:

   • repo: This is the name of your GitHub repository IAC-REPO-NAME that you created in Create a GitHub repository.

   • projects: A list / array of Google Cloud projects IDs that this IaC GitHub repository maps to. In this tutorial, it is your test project.

   • stub: Recommender generates IAM recommendations when a subset of permissions for a role have not been used for 60 days and VM sizing recommendations follow a similar pattern. For the purposes of testing this pipeline on demand, stub can be passed in as true so that the pipeline is tested using the sample Recommender payloads provided in the repository that you cloned for this tutorial.

   gcloud beta scheduler jobs create http recommender-iam-scheduler \
     --project $BUILD_PROJECT_ID \
     --time-zone "America/Los_Angeles" \
     --schedule="0 */3 * * *" \
     --uri=$RECOMMENDER_ROUTE_TO_INVOKE_IAM \
     --description="Scheduler job to invoke recommendation pipeline" \
     --oidc-service-account-email="recommender-scheduler-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com" \
     --headers="Content-Type=application/json" \
     --http-method="POST" \
     --message-body="{ \"repo\": \"IAC-REPO-NAME\", \"projects\": [\"$TEST_PROJECT_ID\"], \"location\": \"global\", \"stub\": true }"

Additional steps

Cloud Build publishes build information to a Pub/Sub topic called cloud-builds that was automatically created when you enabled the Cloud Build API in your build project.

1. Run the following command to verify the cloud-builds topic exists in your build project:

   gcloud pubsub topics describe cloud-builds
   

   If the topic exists, you will see the following output, where BUILD-PROJECT-ID is your build project ID:

   name: projects/BUILD-PROJECT-ID/topics/cloud-builds
   

   If you receive an error message saying the resource was not found, follow the instructions for subscribing to build notifications, to create the topic manually.

2. Create a service account that Pub/Sub uses to invoke the recommender-parser service endpoint.

   gcloud beta iam service-accounts create recommender-ci-subscription-sa \
     --description "Service Account used by Cloud Pub/Sub to push Cloud Build events to the recommender-parser service" \
     --display-name "recommender-ci-subscription-sa" \
     --project $BUILD_PROJECT_ID
   

3. The Pub/Sub service account should be associated with the roles it needs to be able to publish messages and invoke the recommender-parser service.

   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/pubsub.publisher \
     --project $BUILD_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/pubsub.subscriber \
     --project $BUILD_PROJECT_ID
   
   gcloud projects add-iam-policy-binding $BUILD_PROJECT_ID \
     --member serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role roles/run.invoker \
     --project $BUILD_PROJECT_ID
   

4. Add the recommender-ci-subscription-saservice account you created to the recommender-parser service as a member with the invoker role

   gcloud beta run services add-iam-policy-binding recommender-parser \
     --member=serviceAccount:recommender-ci-subscription-sa@$BUILD_PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/run.invoker --region=us-central1
   

5. Navigate to Pub/Sub in Google Cloud console.

6. Click the cloud-builds topic.

7. Click Create Subscription.

8. For Subscription ID, type recommender-service-build-events.

9. For Delivery Type, select Push.

10. For Endpoint, type in your recommender-service URL appended by /ci.

11. Check Enable Authentication.

    1. Select the service account recommender-ci-subscription-sathat you created.
    2. Click Grant in response to the prompt message.

12. Select Acknowledgement deadline as 60 seconds.

13. Keep rest of the defaults.

14. Click Create.

Test the pipeline

Recommender generates IAM recommendations when a subset of permissions for a role have not been used for 60 days. VM sizing recommendations follow a similar pattern. For the purposes of testing this pipeline on demand, you will use the sample recommendations JSON payloads provided in the stub sub-directory provided in the repository you cloned for this tutorial. This enables you to test the pipeline, barring the API calls that the recommender-parser makes to the Recommender API endpoint to update the status of recommendations it has successfully applied.

Alternatively, if you have active recommendations in your Google Cloud projects then you can test the pipeline end-to-end without needing to use stubs. The outcome outlined below is pertinent to when you are using the sample payloads for testing the pipeline. However, the steps to test this pipeline without samples remains the same.

1. In the Google Cloud console, navigate to your test project and review the resource that were created. You should have the following:

   1. A Compute Engine instance called tf-compute-1 with Machine Type g1-small.
   2. A service account named Terraform Recommender Test with the role of editor for your test project.

2. In the Cloud Scheduler console page in your build project, click Run now for the recommender-iam-scheduler job.

3. Click the job to view the logs. You can also view the recommender-parser service logs to get a detailed view of the steps being executed by the service.

4. Once the service completes its run, navigate to your GitHub IAC-REPO-NAME repository. The recommender-parserservice would have generated a pull request for you. Review the modified IaC manifests that comprise this pull request and click Merge Pull Request if you are satisfied with the changes to your IaC manifests.

5. A new commit to the master branch is created when you merge the pull request. This triggers a Cloud Build job that rolls out the modifications to the Google Cloud resources in your test project. Allow a few moments for the Cloud Build job to complete, you can review its status in the Google Cloud console

6. Once the job is complete, navigate to your test project. The sample payloads provided make the following changes to the resources in your test project.

   • Terraform Test service account which earlier has a role of editor when deployed, is changed to viewer.