使用工作建構工具執行管道

事前準備

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Go to project selector

Make sure that billing is enabled for your Google Cloud project.

Enable the Dataflow, Compute Engine, Cloud Logging, Cloud Storage, Google Cloud Storage JSON, and Resource Manager APIs.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Go to project selector

Make sure that billing is enabled for your Google Cloud project.

Enable the Dataflow, Compute Engine, Cloud Logging, Cloud Storage, Google Cloud Storage JSON, and Resource Manager APIs.

Enable the APIs

1. 建立 Cloud Storage bucket：
1. In the Google Cloud console, go to the Cloud Storage Buckets page.

   Go to Buckets

2. Click add_box Create.
3. On the Create a bucket page, enter your bucket information. To go to the next step, click Continue.
   1. For Name your bucket, enter a unique bucket name. Don't include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
   2. In the Choose where to store your data section, do the following:
      1. Select a Location type.
      2. Choose a location where your bucket's data is permanently stored from the Location type drop-down menu.
         • If you select the dual-region location type, you can also choose to enable turbo replication by using the relevant checkbox.
      3. To set up cross-bucket replication, select Add cross-bucket replication via Storage Transfer Service and follow these steps:

         Set up cross-bucket replication

         1. In the Bucket menu, select a bucket.

         2. In the Replication settings section, click Configure to configure settings for the replication job.

            The Configure cross-bucket replication pane appears.

            • To filter objects to replicate by object name prefix, enter a prefix that you want to include or exclude objects from, then click add Add a prefix.
            • To set a storage class for the replicated objects, select a storage class from the Storage class menu. If you skip this step, the replicated objects will use the destination bucket's storage class by default.
            • Click Done.
   3. In the Choose how to store your data section, do the following:
      1. In the Set a default class section, select the following: Standard.
      2. To enable hierarchical namespace, in the Optimize storage for data-intensive workloads section, select Enable hierarchical namespace on this bucket.
   4. In the Choose how to control access to objects section, select whether or not your bucket enforces public access prevention, and select an access control method for your bucket's objects.
   5. In the Choose how to protect object data section, do the following:
      • Select any of the options under Data protection that you want to set for your bucket.
        • To enable soft delete, click the Soft delete policy (For data recovery) checkbox, and specify the number of days you want to retain objects after deletion.
        • To set Object Versioning, click the Object versioning (For version control) checkbox, and specify the maximum number of versions per object and the number of days after which the noncurrent versions expire.
        • To enable the retention policy on objects and buckets, click the Retention (For compliance) checkbox, and then do the following:
          • To enable Object Retention Lock, click the Enable object retention checkbox.
          • To enable Bucket Lock, click the Set bucket retention policy checkbox, and choose a unit of time and a length of time for your retention period.
      • To choose how your object data will be encrypted, expand the Data encryption section (expand_more), and select a Data encryption method.
4. Click Create.

2. 如要完成本快速入門導覽中的步驟，使用者帳戶必須具備 Dataflow 管理員角色和服務帳戶使用者角色。Compute Engine 預設服務帳戶必須具備 Dataflow 工作者角色。 如要在 Google Cloud 控制台中新增必要角色，請按照下列步驟操作：

   1. 前往身分與存取權管理頁面。
      前往「IAM」頁面
   2. 選取專案。
   3. 在包含您使用者帳戶的資料列中，按一下 edit「Edit principal」(編輯主體)。
   4. 按一下「新增其他角色」add，然後在下拉式清單中選取「Dataflow 管理員」。
   5. 按一下「Add another role」(新增其他角色)add，然後在下拉式清單中選取「Service Account User」(服務帳戶使用者)。
   6. 按一下 [儲存]。
   7. 在包含 Compute Engine 預設服務帳戶的資料列中，按一下 edit「Edit principal」(編輯主體)。
   8. 按一下「新增其他角色」add，然後在下拉式清單中選取「Dataflow Worker」。
   9. 按一下「新增其他角色」add，然後在下拉式清單中選取「Storage 物件管理員」。

   10. 按一下 [儲存]。

       如要進一步瞭解如何授予角色，請參閱「使用控制台授予 IAM 角色」。

3. 根據預設，每個新專案一開始都會具備預設網路。如果專案的預設網路已停用或刪除，您必須在專案中建立網路，並為使用者帳戶指派Compute 網路使用者角色 (roles/compute.networkUser)。