使用 GPU 處理 Landsat 衛星影像

本教學課程說明如何在 Dataflow 上使用 GPU 處理 Landsat 8 衛星影像，並將其轉譯為 JPEG 檔案。本教學課程以「使用 GPU 處理 Landsat 衛星影像」範例為基礎。

目標

  • 為 Dataflow 建構支援 GPU 的 TensorFlow Docker 映像檔。
  • 使用 GPU 執行 Dataflow 工作。

費用

本教學課程使用 Google Cloud的計費元件，包括：

  • Cloud Storage
  • Dataflow
  • Artifact Registry

您可以使用 Pricing Calculator 來根據預測使用量產生預估費用。

事前準備

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. Install the Google Cloud CLI.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command: 

    gcloud init

  5. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Dataflow, Cloud Build, and Artifact Registry APIs: 

    gcloud services enable dataflow cloudbuild.googleapis.com artifactregistry.googleapis.com

  8. If you're using a local shell, then create local authentication credentials for your user account: 

    gcloud auth application-default login

    You don't need to do this if you're using Cloud Shell.

    If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.

  9. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser 

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
    • Replace PROJECT_ID with your project ID.

    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.

    • Replace ROLE with each individual role.

  10. Install the Google Cloud CLI.

  11. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  12. To initialize the gcloud CLI, run the following command: 

    gcloud init

  13. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  14. Make sure that billing is enabled for your Google Cloud project.

  15. Enable the Dataflow, Cloud Build, and Artifact Registry APIs: 

    gcloud services enable dataflow cloudbuild.googleapis.com artifactregistry.googleapis.com

  16. If you're using a local shell, then create local authentication credentials for your user account: 

    gcloud auth application-default login

    You don't need to do this if you're using Cloud Shell.

    If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.

  17. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/iam.serviceAccountUser 

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
    • Replace PROJECT_ID with your project ID.

    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.

    • Replace ROLE with each individual role.
    18.

  18. 將角色授予 Compute Engine 預設服務帳戶。針對下列每個 IAM 角色，執行一次下列指令：roles/dataflow.admin、roles/dataflow.worker、roles/bigquery.dataEditor、roles/pubsub.editor、roles/storage.objectAdmin 和 roles/artifactregistry.reader。

    gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" --role=SERVICE_ACCOUNT_ROLE
    • 將 PROJECT_ID 替換為您的專案 ID。
    • 將 PROJECT_NUMBER 替換為專案編號。如要找出專案編號，請參閱「識別專案」。
    • 將 SERVICE_ACCOUNT_ROLE 替換為各個角色。
  19. 如要儲存本教學課程的輸出 JPEG 圖片檔，請建立 Cloud Storage 值區：
    1. In the Google Cloud console, go to the Cloud Storage Buckets page.

      Go to Buckets

    2. Click Create.
    3. On the Create a bucket page, enter your bucket information. To go to the next step, click Continue.
      1. For Name your bucket, enter a unique bucket name. Don't include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
      2. In the Choose where to store your data section, do the following:
        1. Select a Location type.
        2. Choose a location where your bucket's data is permanently stored from the Location type drop-down menu.
        3. To set up cross-bucket replication, select Add cross-bucket replication via Storage Transfer Service and follow these steps:

          Set up cross-bucket replication

          1. In the Bucket menu, select a bucket.

          2. In the Replication settings section, click Configure to configure settings for the replication job.

            The Configure cross-bucket replication pane appears.

            • To filter objects to replicate by object name prefix, enter a prefix that you want to include or exclude objects from, then click Add a prefix.
            • To set a storage class for the replicated objects, select a storage class from the Storage class menu. If you skip this step, the replicated objects will use the destination bucket's storage class by default.
            • Click Done.
      3. In the Choose how to store your data section, do the following:
        1. In the Set a default class section, select the following: Standard.
        2. To enable hierarchical namespace, in the Optimize storage for data-intensive workloads section, select Enable hierarchical namespace on this bucket.
      4. In the Choose how to control access to objects section, select whether or not your bucket enforces public access prevention, and select an access control method for your bucket's objects.
      5. In the Choose how to protect object data section, do the following:
        • Select any of the options under Data protection that you want to set for your bucket.
          • To enable soft delete, click the Soft delete policy (For data recovery) checkbox, and specify the number of days you want to retain objects after deletion.
          • To set Object Versioning, click the Object versioning (For version control) checkbox, and specify the maximum number of versions per object and the number of days after which the noncurrent versions expire.
          • To enable the retention policy on objects and buckets, click the Retention (For compliance) checkbox, and then do the following:
            • To enable Object Retention Lock, click the Enable object retention checkbox.
            • To enable Bucket Lock, click the Set bucket retention policy checkbox, and choose a unit of time and a length of time for your retention period.
        • To choose how your object data will be encrypted, expand the Data encryption section (), and select a Data encryption method.
    4. Click Create.

準備工作環境

下載入門檔案，然後建立 Artifact Registry 存放區。

下載範例檔案

下載入門檔案，然後變更目錄。

  1. 複製 python-docs-samples 存放區。

    git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

  2. 前往範例程式碼目錄。

    cd python-docs-samples/dataflow/gpu-examples/tensorflow-landsat

設定 Artifact Registry

建立 Artifact Registry 存放區，以便上傳構件。每個存放區只能包含單一支援格式的構件。

所有存放區內容都會使用 Google-owned and Google-managed encryption keys 或客戶管理的加密金鑰加密。Artifact Registry 預設會使用Google-owned and Google-managed encryption keys ，不需要為這個選項進行任何設定。

您必須至少具備存放區的Artifact Registry 寫入者存取權。

執行下列指令來建立新的存放區。這個指令會使用 --async 旗標並立即傳回，不會等待進行中的作業完成。

gcloud artifacts repositories create REPOSITORY \
    --repository-format=docker \
    --location=LOCATION \
    --async

將 REPOSITORY 替換為存放區名稱。專案中每個存放區位置的存放區名稱不得重複。

在推送或提取映像檔前，請先設定 Docker，驗證傳送至 Artifact Registry 的要求。如要為 Docker 存放區設定驗證機制，請執行下列指令：

gcloud auth configure-docker LOCATION-docker.pkg.dev

這個指令會更新 Docker 設定。您現在可以在 Google Cloud 專案中連結 Artifact Registry，以推送映像檔。

建構 Docker 映像檔

Cloud Build 可讓您使用 Dockerfile 建構 Docker 映像檔，並將其儲存至 Artifact Registry，供其他Google Cloud 產品存取。

使用 build.yaml 設定檔建構容器映像檔。

gcloud builds submit --config build.yaml

使用 GPU 執行 Dataflow 工作

以下程式碼區塊示範如何使用 GPU 啟動這個 Dataflow 管道。

我們使用 run.yaml 設定檔執行 Dataflow 管道。

export PROJECT=PROJECT_NAME
export BUCKET=BUCKET_NAME

export JOB_NAME="satellite-images-$(date +%Y%m%d-%H%M%S)"
export OUTPUT_PATH="gs://$BUCKET/samples/dataflow/landsat/output-images/"
export REGION="us-central1"
export GPU_TYPE="nvidia-tesla-t4"

gcloud builds submit \
    --config run.yaml \
    --substitutions _JOB_NAME=$JOB_NAME,_OUTPUT_PATH=$OUTPUT_PATH,_REGION=$REGION,_GPU_TYPE=$GPU_TYPE \
    --no-source

更改下列內容：

  • PROJECT_NAME：專案名稱 Google Cloud
  • BUCKET_NAME：Cloud Storage 值區名稱 (不含 gs:// 前置字串)

執行這個管道後，請等待指令執行完畢。如果退出殼層，您設定的環境變數可能會遺失。

為避免在多個工作站程序之間共用 GPU，這個範例使用 1 個 vCPU 的機器類型。使用 13 GB 的擴充記憶體，即可滿足管道的記憶體需求。詳情請參閱GPU 和工作站並行。

查看結果

tensorflow-landsat/main.py 中的管道會處理 Landsat 8 衛星影像，並將其算繪為 JPEG 檔案。請按照下列步驟查看這些檔案。

  1. 使用 Google Cloud CLI 列出輸出 JPEG 檔案的詳細資料。

    gcloud storage ls "gs://$BUCKET/samples/dataflow/landsat/" --long --readable-sizes

  2. 將檔案複製到本機目錄。

    mkdir outputs
gcloud storage cp "gs://$BUCKET/samples/dataflow/landsat/*" outputs/

  3. 使用您選擇的圖片檢視器開啟這些圖片檔案。

清除所用資源

如要避免系統向您的 Google Cloud 帳戶收取本教學課程中所用資源的相關費用，請刪除含有該項資源的專案，或者保留專案但刪除個別資源。

刪除專案

如要避免付費，最簡單的方法就是刪除您為了本教學課程所建立的專案。

如要刪除專案：

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

後續步驟