Step 1: Create a cluster

GKE

Create a cluster on GKE

These steps explain how to configure and create a GKE cluster in your Google Cloud project.

Apigee recommends creating a regional clusters rather than a zonal clusters. If you are unfamiliar with the distinction between regions and zones, see Regions and zones. The available regions are listed in Available regions and zones. Just be aware that, for example, us-west1 is a valid region name, while us-west1-a is a zone in the region.

1. Make sure you are using a version of GKE that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, make sure they are synchronized with NTP across all regions.
3. (GKE private clusters only), If you are creating a private cluster on GKE, add a firewall rule to allow port 9443 for communication between GKE master nodes and GKE worker nodes and to allow the GKE masters to access Apigee mutating webhooks. Follow the procedure in Adding firewall rules for specific use cases in the Google Kubernetes Engine documentation. For more information see Private clusters in GKE.

   You do not need to add this rule if you are creating a standard or public cluster.

4. Create a standard cluster by following the instructions at Create a regional cluster with a multi-zone node pool. It's okay to create the cluster with just the default node pool. You will configure and create the required Apigee hybrid node pools in the next step.

   Go to the next step only after the cluster creation completes successfully.

5. Create two node pools by following the instructions in Add and manage node pools. Be sure to configure the node pools with the minimum requirements listed in the table below.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • About node pools
   • Cassandra network requirements

   Minimum node pool requirements

   Be sure to satisfy these minimum requirements when creating the node pools. If using the Cloud console, be sure to configure both the Node pool details and Nodes sections.

   Node pool name	Description	Minimum nodes	Minimum machine type
   apigee-data	A stateful node pool used for the Cassandra database.	1 per zone (3 per region)	e2-standard-4 (4 vCPU, 16 GB memory)
   apigee-runtime	A stateless node pool used by the runtime message processor.	1 per zone (3 per region)	e2-standard-4 (4 vCPU, 16 GB memory)

   For more details about node pool configuration see Configure dedicated node pools.

6. (Optional) If you wish, you can delete the default node pool. See Delete a node pool.
7. If you haven't already, create the following environment variables. These variables are used in the gcloud commands that follow.

   Linux / MacOS

   export CLUSTER_NAME="YOUR_CLUSTER_NAME"
   export CLUSTER_LOCATION="YOUR_CLUSTER_LOCATION"
   export PROJECT_ID="YOUR_PROJECT_ID"

   Windows

   set CLUSTER_NAME="YOUR_CLUSTER_NAME"
   set CLUSTER_LOCATION=YOUR_CLUSTER_LOCATION
   set PROJECT_ID=YOUR_PROJECT_ID

   Where:

   • CLUSTER_NAME: The name of your cluster.
   • CLUSTER_LOCATION: The region in which you created your cluster.
   • PROJECT_ID: Your Google Cloud project ID.
8. Verify the node pool configurations:

   Regional clusters

   gcloud container node-pools list \
   --cluster=${CLUSTER_NAME} \
   --region=${CLUSTER_LOCATION} \
   --project=${PROJECT_ID}

   Zonal clusters

   gcloud container node-pools list \
   --cluster=${CLUSTER_NAME} \
   --zone=${CLUSTER_LOCATION} \
   --project=${PROJECT_ID}

9. Make sure your cluster is set as the default cluster for kubectl by getting the
   gcloud credentials of the cluster you just created:

   Regional clusters

     gcloud container clusters get-credentials ${CLUSTER_NAME} \
       --region ${CLUSTER_LOCATION} \
       --project ${PROJECT_ID}

   Zonal clusters

     gcloud container clusters get-credentials ${CLUSTER_NAME} \
       --zone ${CLUSTER_LOCATION} \
       --project ${PROJECT_ID}

   See Set a default cluster for kubectl commands.

10. Configure persistent solid state disk (SSD) storage for Cassandra. We do not support using local SSDs. For more information, see Change the default storage class in the Kubernetes documentation.

    1. Get the name of the current default StorageClass:

       kubectl get sc

       For example:

       kubectl get sc
         NAME                    PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
         premium-rwo             pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h
         standard                kubernetes.io/gce-pd    Delete          Immediate              true                   15h
         standard-rwo (default)  pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h

    2. Describe the StorageClass named standard-rwo. Note that its type is pd-balanced:

       kubectl describe sc standard-rwo

       For example:

       kubectl describe sc standard-rwo

       Name:                  standard-rwo
         IsDefaultClass:        Yes
         Annotations:           components.gke.io/layer=addon,storageclass.kubernetes.io/is-default-class=false
         Provisioner:           pd.csi.storage.gke.io
         Parameters:            type=pd-balanced
         AllowVolumeExpansion:  True
         MountOptions:          <none>
         ReclaimPolicy:         Delete
         VolumeBindingMode:     WaitForFirstConsumer
         Events:                <none>

    3. Create a new file called storageclass.yaml.
    4. Add this code to the file. Note that the name of the new storage class is apigee-sc. You can use any name you like. Also, note that the storage type is pd-ssd:

       ---
       kind: StorageClass
       apiVersion: storage.k8s.io/v1
       metadata:
         name: "apigee-sc"
       provisioner: kubernetes.io/gce-pd
       parameters:
         type: pd-ssd
         replication-type: none
       volumeBindingMode: WaitForFirstConsumer
       allowVolumeExpansion: true

    5. Apply the new StorageClass to your Kubernetes cluster:

       kubectl apply -f storageclass.yaml

    6. Execute the following two commands to change the default StorageClass:

       kubectl patch storageclass standard-rwo \
       -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

       kubectl patch storageclass apigee-sc \
       -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

    7. Execute this command to verify that the new default StorageClass is called apigee-sc:

       kubectl get sc

       For example:

       kubectl get sc
         NAME                  PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
         apigee-sc (default)   kubernetes.io/gce-pd    Delete          WaitForFirstConsumer   true                   14h
         premium-rwo           pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h
         standard              kubernetes.io/gce-pd    Delete          Immediate              true                   15h
         standard-rwo          pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h

11. Enable workload identity for the cluster. Workload Identity allows workloads in your GKE clusters to impersonate Identity and Access Management (IAM) service accounts to access Google Cloud services. This operation can take up to 30 minutes:

    Regional clusters

    gcloud container clusters update ${CLUSTER_NAME} \
      --workload-pool=${PROJECT_ID}.svc.id.goog \
      --project ${PROJECT_ID} \
      --region ${CLUSTER_LOCATION}

    Zonal clusters

    gcloud container clusters update ${CLUSTER_NAME} \
      --workload-pool=${PROJECT_ID}.svc.id.goog \
      --zone ${CLUSTER_LOCATION} \
      --project ${PROJECT_ID}

12. Verify whether Workload identity is successfully enabled with the following command;

    Regional clusters

    gcloud container clusters describe ${CLUSTER_NAME} \
      --project ${PROJECT_ID} \
      --region ${CLUSTER_LOCATION} | grep -i "workload"

    Zonal clusters

    gcloud container clusters describe ${CLUSTER_NAME} \
      --zone ${CLUSTER_LOCATION} \
      --project ${PROJECT_ID} | grep -i "workload"

When you have a cluster installed and running, go to the next step.

GKE on-prem

Create a cluster on GKE on-prem

These steps explain how to configure and create a GKE on-prem cluster for Apigee hybrid.

1. Make sure you are using a version of Google Distributed Cloud that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. Create the cluster by following the instructions at Create basic clusters. It's okay to create the cluster with just the default node pool. You will configure and create the required Apigee hybrid node pools in the next step.

   Go to the next step only after the cluster creation completes successfully.

4. Create two node pools by following the instructions in Creating and managing node pools. Configure the node pools with the minimum requirements listed in the table below.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • About node pools
   • Cassandra network requirements

   Minimum node pool requirements

   Be sure to satisfy these minimum requirements when creating the node pools.

   Node pool name	Description	Minimum nodes	Minimum machine type
   apigee-data	A stateful node pool used for the Cassandra database.	1 per zone (3 per region)	e2-standard-4 (4 vCPU, 16 GB memory)
   apigee-runtime	A stateless node pool used by the runtime message processor.	1 per zone (3 per region)	e2-standard-4 (4 vCPU, 16 GB memory)

   For more details about node pool configuration see Configure dedicated node pools.

5. (Optional) If you wish, you can delete the default node pool. See Delete a node pool.

6. Configure persistent solid state disk (SSD) storage for Cassandra. We do not support using local SSDs. For more information, see Change the default storage class in the Kubernetes documentation.

   1. Get the name of the current default StorageClass:

      kubectl get sc

      For example:

      kubectl get sc
        NAME                    PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
        premium-rwo             pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h
        standard                kubernetes.io/gce-pd    Delete          Immediate              true                   15h
        standard-rwo (default)  pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h

   2. Describe the StorageClass named standard-rwo. Note that its type is pd-balanced:

      kubectl describe sc standard-rwo

      For example:

      kubectl describe sc standard-rwo

      Name:                  standard-rwo
      IsDefaultClass:        Yes
      Annotations:           components.gke.io/layer=addon,storageclass.kubernetes.io/is-default-class=false
      Provisioner:           pd.csi.storage.gke.io
      Parameters:            type=pd-balanced
      AllowVolumeExpansion:  True
      MountOptions:          <none>
      ReclaimPolicy:         Delete
      VolumeBindingMode:     WaitForFirstConsumer
      Events:                <none>

   3. Create a new file called storageclass.yaml.
   4. Add this code to the file. Note that the name of the new storage class is apigee-sc. You can use any name you like. Also, note that the storage type is pd-ssd:

      ---
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: "apigee-sc"
      provisioner: kubernetes.io/gce-pd
      parameters:
        type: pd-ssd
        replication-type: none
      volumeBindingMode: WaitForFirstConsumer
      allowVolumeExpansion: true

   5. Apply the new StorageClass to your Kubernetes cluster:

      kubectl apply -f storageclass.yaml

   6. Execute the following two commands to change the default StorageClass:

      kubectl patch storageclass standard-rwo \
        -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

      kubectl patch storageclass apigee-sc \
        -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

   7. Execute this command to verify that the new default StorageClass is called apigee-sc:

      kubectl get sc

      For example:

      kubectl get sc
        NAME                  PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
        apigee-sc (default)   kubernetes.io/gce-pd    Delete          WaitForFirstConsumer   true                   14h
        premium-rwo           pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h
        standard              kubernetes.io/gce-pd    Delete          Immediate              true                   15h
        standard-rwo          pd.csi.storage.gke.io   Delete          WaitForFirstConsumer   true                   15h

When you have a cluster installed and running, go to the next step.

Google Distributed Cloud on bare metal

Create a cluster on Google Distributed Cloud on bare metal

These steps explain how to configure and create a cluster for Apigee hybrid on Anthos on bare metal. Anthos on bare metal lets you run Kubernetes clusters directly on your own machine resources.

1. Make sure you are using a version of Google Distributed Cloud on Bare Metal that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. Review the Installation prerequisites overview and Creating clusters: overview.
4. Create the cluster with two node pools configured as described below:
   • Follow the cluster creation instructions Create basic clusters in the Google Distributed Cloud on bare metal documentation.
   • Create a minimum of two node pools. Configure the node pools with the minimum requirements listed in the table below.

     What you need to know about node pools

     We recommend you review the following information about node pools and node pool configuration before continuing:

     • Configure dedicated node pools
     • About node pools
     • Cassandra network requirements

     The minimum configurations for your cluster are:

     Configuration	Stateful node pool	Stateless node pool
     Purpose	A stateful node pool used for the Cassandra database.	A stateless node pool used by the runtime message processor.
     Label name	apigee-data	apigee-runtime
     Number of nodes	1 per zone (3 per region)	1 per zone (3 per region)
     CPU	4	4
     RAM	15	15
     Storage	dynamic	Managed with the ApigeeDeployment CRD
     Minimum disk IOPS	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.
     Network bandwidth for each machine instance type	1 Gbps	1 Gbps

     For more details on minimum cluster configuration see: Minimum cluster configurations

When you have a cluster installed and running, go to the next step.

AKS

Create a cluster on AKS

These steps explain how to configure and create a cluster for Apigee hybrid on AKS.

1. Make sure you are using a version of AKS that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. Create the cluster using either the Azure CLI or Azure Portal, and create two node pools as described below.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • Configure dedicated node pools
   • About node pools
   • Cassandra network requirements

   The minimum configurations for your cluster are:

   Configuration	Stateful node pool	Stateless node pool
   Purpose	A stateful node pool used for the Cassandra database.	A stateless node pool used by the runtime message processor.
   Label name	apigee-data	apigee-runtime
   Number of nodes	1 per zone (3 per region)	1 per zone (3 per region)
   CPU	4	4
   RAM	15	15
   Storage	dynamic	Managed with the ApigeeDeployment CRD
   Minimum disk IOPS	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.
   Network bandwidth for each machine instance type	1 Gbps	1 Gbps

   For more details on minimum cluster configuration see: Minimum cluster configurations

When you have a cluster installed and running, go to the next step.

EKS

Create a cluster on EKS

These steps explain how to configure and create a cluster for Apigee hybrid on EKS.

1. Make sure you are using a version of EKS that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. If you are using Kubernetes version 1.24 or newer, make sure you have installed the Kubernetes CSI driver for Amazon EBS.
4. Use the following instructions to create a user cluster, and create two node pools as described below.
   • EKS overview documentation.
   • Creating an Amazon EKS cluster.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • Configure dedicated node pools
   • About node pools
   • Cassandra network requirements

   The minimum configurations for your cluster are:

   Configuration	Stateful node pool	Stateless node pool
   Purpose	A stateful node pool used for the Cassandra database.	A stateless node pool used by the runtime message processor.
   Label name	apigee-data	apigee-runtime
   Number of nodes	1 per zone (3 per region)	1 per zone (3 per region)
   CPU	4	4
   RAM	15	15
   Storage	dynamic	Managed with the ApigeeDeployment CRD
   Minimum disk IOPS	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.
   Network bandwidth for each machine instance type	1 Gbps	1 Gbps

   For more details on minimum cluster configuration see: Minimum cluster configurations

When you have a cluster installed and running, go to the next step.

GKE on AWS

Create a cluster on GKE on AWS

These steps explain how to configure and create a cluster for Apigee hybrid on GKE on AWS.

1. Make sure you are using a version of GKE that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. Use the following instructions to create a user cluster, and create two node pools as described below.
   • GKE on AWS overview documentation.
   • Deploy an application on GKE on AWS.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • Configure dedicated node pools
   • About node pools
   • Cassandra network requirements

   The minimum configurations for your cluster are:

   Configuration	Stateful node pool	Stateless node pool
   Purpose	A stateful node pool used for the Cassandra database.	A stateless node pool used by the runtime message processor.
   Label name	apigee-data	apigee-runtime
   Number of nodes	1 per zone (3 per region)	1 per zone (3 per region)
   CPU	4	4
   RAM	15	15
   Storage	dynamic	Managed with the ApigeeDeployment CRD
   Minimum disk IOPS	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.
   Network bandwidth for each machine instance type	1 Gbps	1 Gbps

   For more details on minimum cluster configuration see: Minimum cluster configurations

When you have a cluster installed and running, go to the next step.

OpenShift

Create a cluster on OpenShift

These steps explain how to configure and create a cluster for Apigee hybrid on OpenShift.

1. Make sure you are using a version of OpenShift that is supported for hybrid version 1.11.2. See Apigee hybrid supported platforms and versions.
2. Ensure the clocks on all nodes and application servers are synchronized with Network Time Protocol (NTP), as explained in the Prerequisites. The Cassandra database relies on Network Time Protocol (NTP) synchronization to maintain data consistency. If you plan to install hybrid into multiple regions, be sure they are synchronized with NTP across all regions.
3. Build the OpenShift cluster to deploy on the runtime plane, install Apigee on your OpenShift user cluster, and create two node pools.

   As part of the OpenShift install, install and configure the oc CLI tool. See Getting started with the OpenShift CLI in the OpenShift documentation.

   What you need to know about node pools

   We recommend you review the following information about node pools and node pool configuration before continuing:

   • Configure dedicated node pools
   • About node pools
   • Cassandra network requirements

   The minimum configurations for your cluster are:

   Configuration	Stateful node pool	Stateless node pool
   Purpose	A stateful node pool used for the Cassandra database.	A stateless node pool used by the runtime message processor.
   Label name	apigee-data	apigee-runtime
   Number of nodes	1 per zone (3 per region)	1 per zone (3 per region)
   CPU	4	4
   RAM	15	15
   Storage	dynamic	Managed with the ApigeeDeployment CRD
   Minimum disk IOPS	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.	2000 IOPS with SAN or directly attached storage. NFS is not recommended even if it can support the required IOPS.
   Network bandwidth for each machine instance type	1 Gbps	1 Gbps

   For more details on minimum cluster configuration see: Minimum cluster configurations

When you have installed a cluster, go to the next step.