Jump Start Solution: Large data sharing Go web app

Last reviewed 2023-06-27 UTC

This guide helps you understand, deploy, and use the Large data sharing Go web app Jump Start Solution. This solution demonstrates how you might architect an application that can serve large quantities of files.

If you want to use Java instead of Go, see the Large data sharing Java web app Jump Start Solution.

This document is intended for developers who have some background with building a web application. It assumes that you're familiar with the Go programming language and basic cloud concepts, though not necessarily Google Cloud. Experience with Terraform is helpful.

Objectives

  • Understand the large data sharing user pattern.
  • Store and serve objects in a cloud-based object storage service.
  • Deploy an application that uses a distributed delivery network to serve many objects.
  • Secure objects at each layer for least privilege access.

Products used

The solution uses the following Google Cloud products:

  • Cloud CDN (Content Delivery Network): Fast, reliable web content and video delivery at global scale.
  • Cloud Storage: An enterprise-ready service that provides low-cost, no-limit object storage for diverse data types. Data is accessible from within and outside of Google Cloud and is replicated geo-redundantly.
  • Cloud Run: A fully-managed service that lets you build and deploy serverless containerized apps. Google Cloud handles scaling and other infrastructure tasks so that you can focus on the business logic of your code.
  • Cloud Load Balancing: A service that provides high-performance, scalable load balancing on Google Cloud.
  • Firestore: A fully managed, scalable, serverless document database.

For information about how these products are configured and how they interact, see the next section.

Architecture

This solution deploys a ready-to-use, fully-implemented image-sharing app. The following diagram shows the architecture of the Google Cloud resources that the solution deploys:

Architecture diagram of an image sharing app that uses Cloud CDN to serve files.

Request flow

The following is the request processing flow of the image sharing app. The steps in the flow are numbered as shown in the preceding architecture diagram.

  1. Requests from users to upload files are routed to the front end using a load balancer.
  2. The frontend routes these requests to the backend which uses Cloud Run.
  3. The backend stores the files in Cloud Storage and metadata about the files in Firestore.
  4. If there is a Cloud CDN cache hit, requests from users to view a file are routed to the Cloud CDN.
  5. If there is a Cloud CDN cache miss, requests from users to view a file are routed to Cloud Storage.

Components and configuration

The architecture includes the following components:

  • An image sharing web app composed of a frontend and a backend running on Cloud Run. The app frontend uses Angular. The app backend is written in Go.
  • A load balancer created using Cloud Load Balancing routes incoming requests to the appropriate resource.
  • A Cloud Storage bucket that stores image files.
  • Cloud CDN caches images from the Cloud Storage bucket and accelerates content delivery.
  • A Firestore database to store file metadata.

Cost

For an estimate of the cost of the Google Cloud resources that the large data sharing Go web app solution uses, see the precalculated estimate in the Google Cloud Pricing Calculator.

Use the estimate as a starting point to calculate the cost of your deployment. You can modify the estimate to reflect any configuration changes that you plan to make for the resources that are used in the solution.

The precalculated estimate is based on assumptions for certain factors, including the following:

  • The Google Cloud locations where the resources are deployed.
  • The amount of time that the resources are used.

Deploy the solution

This section guides you through the process of deploying the solution.

Create or choose a Google Cloud project

When you deploy the solution, you choose the Google Cloud project where the resources are deployed. When you're deciding whether to use an existing project or to create a new project, consider the following factors:

  • If you create a project for the solution, then when you no longer need the deployment, you can delete the project and avoid continued billing. If you use an existing project, you must delete the deployment when you no longer need it.
  • Using a new project can help avoid conflicts with previously provisioned resources, such as resources that are used for production workloads.

If you want to deploy the solution in a new project, create the project before you begin the deployment.

To create a project, complete the following steps:

  1. In the Google Cloud console, go to the project selector page.

    Go to project selector

  2. To begin creating a Google Cloud project, click Create project.

  3. Name your project. Make a note of your generated project ID.

  4. Edit the other fields as needed.

  5. To create the project, click Create.

Get the required IAM permissions

To start the deployment process, you need the Identity and Access Management (IAM) permissions that are listed in the following table. If you have the roles/owner basic role for the project in which you plan to deploy the solution, then you already have all the necessary permissions. If you don't have the roles/owner role, then ask your administrator to grant these permissions (or the roles that include these permissions) to you.

IAM permission required Predefined role that includes the required permissions

serviceusage.services.enable

Service Usage Admin
(roles/serviceusage.serviceUsageAdmin)

iam.serviceAccounts.create

Service Account Admin
(roles/iam.serviceAccountAdmin)

resourcemanager.projects.setIamPolicy

Project IAM Admin
(roles/resourcemanager.projectIamAdmin)
config.deployments.create
config.deployments.list
Cloud Infrastructure Manager Admin
(roles/config.admin)

Service account created for the solution

If you start the deployment process through the console, Google creates a service account to deploy the solution on your behalf (and to delete the deployment later if you choose). This service account is assigned certain IAM permissions temporarily; that is, the permissions are revoked automatically after the solution deployment and deletion operations are completed. Google recommends that after you delete the deployment, you delete the service account, as described later in this guide.

View the roles that are assigned to the service account

These roles are listed here in case an administrator of your Google Cloud project or organization needs this information.

  • roles/storage.admin
  • roles/datastore.owner
  • roles/appengine.appAdmin
  • roles/compute.admin
  • roles/compute.networkAdmin
  • roles/cloudtrace.admin
  • roles/iam.serviceAccountUser
  • roles/run.admin
  • roles/monitoring.admin
  • roles/vpcaccess.admin

Choose a deployment method

To help you deploy this solution with minimal effort, a Terraform configuration is provided in GitHub. The Terraform configuration defines all the Google Cloud resources that are required for the solution.

You can deploy the solution by using one of the following methods:

  • Through the console: Use this method if you want to try the solution with the default configuration and see how it works. Cloud Build deploys all the resources that are required for the solution. When you no longer need the deployed solution, you can delete it through the console. Any resources that you create after you deploy the solution might need to be deleted separately.

    To use this deployment method, follow the instructions in Deploy through the console.

  • Using the Terraform CLI: Use this method if you want to customize the solution or if you want to automate the provisioning and management of the resources by using the infrastructure as code (IaC) approach. Download the Terraform configuration from GitHub, optionally customize the code as necessary, and then deploy the solution by using the Terraform CLI. After you deploy the solution, you can continue to use Terraform to manage the solution.

    To use this deployment method, follow the instructions in Deploy using the Terraform CLI.

Deploy through the console

Complete the following steps to deploy the preconfigured solution.

  1. In the Google Cloud Jump Start Solutions catalog, go to the Large data sharing Go web app solution.

    Go to the Large data sharing Go web app solution

  2. Review the information that's provided on the page, such as the estimated cost of the solution and the estimated deployment time.

  3. When you're ready to start deploying the solution, click Deploy.

    A step-by-step interactive guide is displayed.

  4. Complete the steps in the interactive guide.

    Note the name that you enter for the deployment. This name is required later when you delete the deployment.

    When you click Deploy, the Solution deployments page is displayed. The Status field on this page shows Deploying.

  5. Wait for the solution to be deployed.

    If the deployment fails, the Status field shows Failed. You can use the Cloud Build log to diagnose the errors. For more information, see Errors when deploying through the console.

    After the deployment is completed, the Status field changes to Deployed.

    1. To view and use the solution, return to the Solution deployments page in the console.
      1. Click the Actions menu.
      2. Select View web app to open the image sharing app.

When you no longer need the solution, you can delete the deployment to avoid continued billing for the Google Cloud resources. For more information, see Delete the deployment.

Deploy using the Terraform CLI

This section describes how you can customize the solution or automate the provisioning and management of the solution by using the Terraform CLI. Solutions that you deploy by using the Terraform CLI are not displayed in the Solution deployments page in the Google Cloud console.

Set up the Terraform client

You can run Terraform either in Cloud Shell or on your local host. This guide describes how to run Terraform in Cloud Shell, which has Terraform preinstalled and configured to authenticate with Google Cloud.

The Terraform code for this solution is available in a GitHub repository.

  1. Clone the GitHub repository to Cloud Shell.

    Open in Cloud Shell

    A prompt is displayed to confirm downloading the GitHub repository to Cloud Shell.

  2. Click Confirm.

    Cloud Shell is launched in a separate browser tab, and the Terraform code is downloaded to the $HOME/cloudshell_open directory of your Cloud Shell environment.

  3. In Cloud Shell, check whether the current working directory is $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra. This is the directory that contains the Terraform configuration files for the solution. If you need to change to that directory, run the following command:

    cd $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra
    
  4. Initialize Terraform by running the following command:

    terraform init
    

    Wait until you see the following message:

    Terraform has been successfully initialized!
    

Configure the Terraform variables

The Terraform code that you downloaded includes variables that you can use to customize the deployment based on your requirements. For example, you can specify the Google Cloud project and the region where you want the solution to be deployed.

  1. Make sure that the current working directory is $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra. If it isn't, go to that directory.

  2. In the same directory, create a text file named terraform.tfvars.

  3. In the terraform.tfvars file, copy the following code snippet, and set values for the required variables.

    • Follow the instructions that are provided as comments in the code snippet.
    • This code snippet includes only the variables for which you must set values. The Terraform configuration includes other variables that have default values. To review all the variables and the default values, see the variables.tf file that's available in the $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra directory.
    • Make sure that each value that you set in the terraform.tfvars file matches the variable type as declared in the variables.tf file. For example, if the type that’s defined for a variable in the variables.tf file is bool, then you must specify true or false as the value of that variable in the terraform.tfvars file.
    # This is an example of the terraform.tfvars file.
    # The values in this file must match the variable types declared in variables.tf.
    # The values in this file override any defaults in variables.tf.
    
    # ID of the project in which you want to deploy the solution
    project_id = "PROJECT_ID"
    
    

    For information about the project_id, see Identifying projects

Validate and review the Terraform configuration

  1. Make sure that the current working directory is $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra. If it isn't, go to that directory.

  2. Verify that the Terraform configuration has no errors:

    terraform validate
    

    If the command returns any errors, make the required corrections in the configuration and then run the terraform validate command again. Repeat this step until the command returns the following message:

    Success! The configuration is valid.
    
  3. Review the resources that are defined in the configuration:

    terraform plan
    
  4. If you didn't create the terraform.tfvars file as described earlier, Terraform prompts you to enter values for the variables that don't have default values. Enter the required values.

    The output of the terraform plan command is a list of the resources that Terraform provisions when you apply the configuration.

    If you want to make any changes, edit the configuration and then run the terraform validate and terraform plan commands again.

Provision the resources

When no further changes are necessary in the Terraform configuration, deploy the resources.

  1. Make sure that the current working directory is $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra. If it isn't, go to that directory.

  2. Apply the Terraform configuration:

    terraform apply
    
  3. If you didn't create the terraform.tfvars file as described earlier, Terraform prompts you to enter values for the variables that don't have default values. Enter the required values.

    Terraform displays a list of the resources that will be created.

  4. When you're prompted to perform the actions, enter yes.

    Terraform displays messages showing the progress of the deployment.

    If the deployment can't be completed, Terraform displays the errors that caused the failure. Review the error messages and update the configuration to fix the errors. Then run the terraform apply command again. For help with troubleshooting Terraform errors, see Errors when deploying the solution using the Terraform CLI.

    After all the resources are created, Terraform displays the following message:

    Apply complete!
    
    Outputs:
    bucket_name = "BUCKET_NAME"
    lb_external_ip = "IP_ADDRESS"
    neos_walkthrough_url = "https://console.cloud.google.com/products/solutions/deployments?walkthrough_id=solutions-in-console--large-data-sharing--large-data-sharing--golang-_tour"
    
  5. To view and use the sample app, open thelb_external_ip in your browser.

When you no longer need the solution, you can delete the deployment to avoid continued billing for the Google Cloud resources. For more information, see Delete the deployment.

Customize the solution

This section provides information that Terraform developers can use to modify the large data sharing Go web app solution in order to meet their own technical and business requirements. The guidance in this section is relevant only if you deploy the solution by using the Terraform CLI.

  1. Verify that the current working directory is $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra. If it isn't, go to that directory:

    cd $HOME/cloudshell_open/terraform-large-data-sharing-golang-webapp/infra
    
  2. Open main.tf and make the required changes, as shown in the examples in the following table:


    Parameter

    Terraform argument in main.tf

    CPU limit for the app backend

    Argument: cloud_run_server.limits.cpu

    Relevant code snippet in main.tf:

    module "cloud_run_server" {
    …
      limits = {
        cpu    = "2000m"
        …
      }
    


    Memory limit for the app backend

    Argument: cloud_run_server.limits.memory

    Relevant code snippet in main.tf:

    module "cloud_run_server" {
    …
      limits = {
        …
        memory = "2Gi"
        …
    


    CPU limit for the app frontend

    Argument: cloud_run_client.limits.cpu

    Relevant code snippet in main.tf:

    module "cloud_run_client" {
    …
      limits = {
        cpu    = "1000m"
        …
      }
    


    Memory limit for the app frontend

    Argument: cloud_run_client.limits.memory

    Relevant code snippet in main.tf:

    module "cloud_run_client" {
    …
    
      limits = {
        …
        memory = "512Mi"
        …
    

  3. Validate and provision the resources using the same steps as in the previous section:

    1. Validate and review the Terraform configuration.

    2. Provision the resources.

Delete the deployment

When you no longer need the solution deployment, to avoid continued billing for the resources that you created, delete the deployment.

Delete the project

If you deployed the solution in a new Google Cloud project, and if you no longer need the project, then delete it by completing the following steps:

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. At the prompt, type the project ID, and then click Shut down.

If you decide to retain the project, then delete the service account that was created for this solution, as described in the next section.

Delete through the console

If you