Migrate Apigee hybrid to Helm from apigeectl

This migration tool assists in migrating an apigeectl-based hybrid cluster to a Helm-based hybrid cluster. This tool performs no actual replacement of any cluster components. It is idempotent and can be run many times on the same cluster, preparing a subset of components and orgs each time.

You can migrate all apigee components at one time, and Helm upgrade operations can be done on a per-component basis after the tool is run.

See Installing and managing Apigee hybrid with Helm charts for information about managing hybrid clusters you have migrated to Helm management with this tool.

Prerequisites

Helm version v3.14.2+.
A working kubeconfig file pointing to a cluster with a working Apigee hybrid 1.12 installation.
Permissions to modify the metadata and annotations on the Kubernetes resources of the hybrid components that you wish to migrate.

Scope

This tool supports the following options at run-time:

Customization of namespace for apigee resources. Default namespace: apigee
Migration of selected hybrid components only. Default: all components are migrated
Migration of a single org only
Migration of a single env only
Migration of a single env-group (apigee-virtualhost) only
Customization of Helm release names for orgs, envs, and env-groups.

Limitations

This tool does not support customizing Helm release names for these hybrid components: apigee-operator, apigee-datastore, apigee-redis, apigee-telemetry, and apigee-ingress-manager.
Interactive customizations made to the Helm release names for orgs, envs, and env-groups do not automatically persist between runs. You can edit the temp file and supply it as an option in later runs.
Env and env-group filtering is done by name only. In some cases this could result in multiple envs and env-groups being migrated on multi-org clusters.

For example, on a multi-org cluster with orgs org1 and org2, if the env prod is present in both orgs, and only --env=prod is specified, both envs will be migrated. If you wish to only migrate a single env, you must also specify an org filter --org=org1 or --org=org2.

Usage

Syntax

apigee-helm-migration [--apigee-namespace=] [--components=] [--dry-run] [--env=org1] [--env-group=org2] [--org=baz] [--kubeconfig=] [-y] [-v] [-f /path/to/releaseNames.yaml]

Generated Helm release names

Every Helm Chart that is deployed on a cluster needs to have a release name, which must be unique within a namespace. Helm release names don't have any naming convention or restriction relative to the chart name. The migration tool generates unique Helm release names for every component.

Chart	Single-org cluster	Multi-org cluster
`apigee-operator`	`operator`	`operator`
`apigee-datastore`	`datastore`	`datastore`
`apigee-telemetry`	`telemetry`	`telemetry`
`apigee-redis`	`redis`	`redis`
`apigee-ingress-manager`	`ingress-manager`	`ingress-manager`
`apigee-org`	`ORG_NAME`	`ORG_NAME`
`apigee-env`	`ENV_NAME[-env[-n]]⁽¹⁾`	`ORG_NAME-ENV_NAME[-env[-n]]`⁽¹⁾
`apigee-virtualhost (envgroup)`	`VH_NAME[-env-group[-n]]`⁽¹⁾	`ORG_NAME-VH_NAME[-env-group[-n]]`⁽¹⁾
⁽¹⁾ Names are suffixed with `-env` or `-env-group` if the generated name conflicts with another generated name. They are further suffixed with `-1` or `-2 …` if they still conflict.

Customizing Helm release names

The migration tool allows interactive customization of Helm release names. If you want to customize the Helm release names non-interactively:

Run the tool once and exit at the first prompt to create a temporary file containing the auto-generated release names. You should see a line like:
```
INFO: 21:32:56 using temp file for release names:  /tmp/apigee-helm-migration-1229129207-releaseNames
```

Move or copy and then edit this file. You can pass this edited file in with the -f option when you run the migration tool. The auto-generated release names look like:

orgs:
  example-apigee-org:
    helmReleaseName: example-apigee-org
    envs:
      prod:
        helmReleaseName: prod
    envGroups:
      prod-envgroup:
        helmReleaseName: prod-envgroup

To customize the Helm release names for an org, env, or envgroup, edit the helmReleaseName field of that object. For example, to rename the org release to custom-org, the env release to custom-env, and the envgroup release to custom-group, the resulting file looks like:

orgs:
  example-apigee-org:
    helmReleaseName: custom-org
    envs:
      prod:
        helmReleaseName: custom-env
    envGroups:
      prod-envgroup:
        helmReleaseName: custom-group

Using custom namespaces

Apigee hybrid runs in two Kubernetes namespaces:

apigee-system: The apigee-operator component always runs in the apigee-system namespace. The Helm migration tool will update the apigee-operator component in the apigee-system namespace regardless of what you specify with the --apigee-namespace flag.
apigee: All hybrid components except apigee-operator run in this namespace. apigee is the default name. You can use any custom namespace for these components.
If you use a custom namespace, you must specify the it with the --apigee-namespace my_custom_namespace flag when you run the Helm migration tool.

You must also add the namespace: my_custom_namespace top-level property to your overrides file.

Directions

Download the migration tool.
Linux
1. Store the latest version number in a variable using the following command:
  export VERSION=$(curl -s "https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/current-version.txt?ignoreCache=1")
2. Check that the variable was populated with a version number using the following command. If you want to use a different version, you can save that in an environment variable instead.
  echo $VERSION
  For example:
  echo $VERSION 1.0.5
3. Download the release package for your operating system using the following command:
  curl -LO https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/${VERSION}/apigee-helm-migration_linux_64.tar.gz
4. Extract the compressed files using the following command:
  tar -xzf apigee-helm-migration_linux_64.tar.gz
Mac OS
1. Store the latest version number in a variable using the following command:
  export VERSION=$(curl -s "https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/current-version.txt?ignoreCache=1")
2. Check that the variable was populated with a version number using the following command. If you want to use a different version, you can save that in an environment variable instead.
  echo $VERSION
  For example:
  echo $VERSION 1.0.5
3. Download the release package for your operating system using the following command:
  curl -LO https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/${VERSION}/apigee-helm-migration_mac_64.tar.gz
4. Extract the compressed files using the following command:
  tar -xzf apigee-helm-migration_mac_64.tar.gz
Windows
1. Store the latest version number in a variable using the following command:
  for /f "tokens=*" %a in ('curl -s https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/current-version.txt') do set VERSION=%a
2. Check that the variable was populated with a version number using the following command. If you want to use a different version, you can save that in an environment variable instead.
  echo %VERSION%
  For example:
  echo %VERSION% 1.0.5
3. Download the release package for your operating system using the following command:
  curl -LO https://storage.googleapis.com/apigee-release/hybrid/apigee-migration-tool/%VERSION%/apigee-helm-migration_windows_64.tar.gz
4. Extract the compressed files using the following command:
  tar xzvf apigee-helm-migration_windows_64.tar.gz

Run the migration tool. If the default options are acceptable, it is sufficient to run the tool without any arguments, and approve the prompt if the generated helm release names are satisfactory. Some example scenarios are given below:

For multi-org updates, it is recommended to run the apigee-helm-migration without options to upgrade all org and environment components.

A simple installation, using the default kubeconfig (~/.kube/config), default apigee namespace, and default Helm release names.

The following command should be sufficient for most, if not all, installations. Helm upgrade operations can be done on a per-component basis after the tool is run.
```
./apigee-helm-migration
```

Migrating all components using a custom namespace:

./apigee-helm-migration --apigee-namespace my_custom_namespace

Migrating only the operator and datastore components:

./apigee-helm-migration --components operator,datastore

  INFO: 00:22:48 using kubeconfig file  /usr/local/google/home/example/.kube/config
  INFO: 00:22:48 namespace for apigee resources:
  INFO: 00:22:48 	 apigee
  INFO: 00:22:48 processing all organizations in cluster
  INFO: 00:22:48 Components to migrate:
  INFO: 00:22:48 	 operator,datastore
  INFO: 00:22:48 dry-run:
  INFO: 00:22:48 	 false
  Continue with patching apigee resources for Helm migration? [y/n]: y
  INFO: 00:22:52 Processing component:  operator
  INFO: 00:22:54 Processing component:  datastore
  INFO: 00:22:55 Migration successful!

Pointing to a specific kubeconfig file and specifying a different name for the apigee namespace.
```
./apigee-helm-migration --kubeconfig /abs/path/to/kubeconf --namespace org1_namespace
```
Migrating all components, but only a single org:
```
./apigee-helm-migration --org=some-test-org
```

The following shows an example output from a successful migration:

INFO: 21:32:55 using kubeconfig file  /usr/local/google/home/example/.kube/config
INFO: 21:32:55 namespace for apigee resources:
INFO: 21:32:55 	 apigee
INFO: 21:32:55 processing all organizations in cluster
INFO: 21:32:55 processing all components
INFO: 21:32:55 dry-run:
INFO: 21:32:55 	 false
INFO: 21:32:55 cluster Apigee information:
INFO: 21:32:55 Apigee Organizations found:
INFO: 21:32:56 	 example-hybrid-dev
INFO: 21:32:56 Apigee Environments found (org: env):
INFO: 21:32:56 	 example-hybrid-dev : prod
INFO: 21:32:56 Apigee EnvGroups(apigeerouteconfigs) found (org: envGroup):
INFO: 21:32:56 	 example-hybrid-dev : prod-envgroup
INFO: 21:32:56 using temp file for release names:  /tmp/apigee-helm-migration-1229129207-releaseNames
INFO: 21:32:56 Helm release names for Apigee orgs/envs/envgroups:
orgs:
example-hybrid-dev:
helmReleaseName: example-hybrid-dev
envs:
  prod:
    helmReleaseName: prod
envGroups:
  prod-envgroup:
    helmReleaseName: prod-envgroup
Make changes to the release names for Apigee orgs/env/envgroups? [y/n]: n
Continue with patching apigee resources for Helm migration? [y/n]: y
INFO: 21:32:59 Processing component:  operator
INFO: 21:33:01 Processing component:  datastore
INFO: 21:33:01 Processing component:  redis
INFO: 21:33:02 Processing component:  ingress-manager
INFO: 21:33:02 Processing component:  telemetry
INFO: 21:33:03 Processing component:  orgs
INFO: 21:33:05 Processing component:  envs
INFO: 21:33:06 Processing component:  env-groups
INFO: 21:33:07 Migration successful!

Potential errors:

Error parsing release names file: Check the passed in release names file.
Resources not found: Check that Apigee hybrid is fully installed, and you have permissions to access the apigee resources.

Configuration properties changes

Make the following changes in your overrides files:

Use metrics.appStackdriverExporter.* instead of metrics.aggregator.*.
Apigee hybrid managed with Helm uses apigeeIngressGateway properties to configure all Apigee ingress gateways in your cluster. ingressGateways properties override the settings in apigeeIngressGateway for the individual named ingress gateway.
See apigeeIngressGateway
- Create an additional apigeeIngressGateway stanza in your overrides file for any ingressGateways properties that are global to all ingress gateways in your cluster. For example:
```
apigeeIngressGateway:
  image:
    url: "PATH_TO_REPOSITORY/apigee-asm-ingress"
    tag: "TAG"
```
  For example:
```
apigeeIngressGateway:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-asm-ingress"
    tag: "1.17.8-asm.20-distroless"
```
- Make sure to include ingressGateways.name. It is required to instantiate your ingress gateway. For example:
The properties to enable Workload Identity have changed:
- gcp.workloadIdentity.enabled replaces gcp.workloadIdentityEnabled.
- If you are using a single service account for all components, you can specify it with: gcp.workloadIdentity.gsa. For example:
```
gcp:
  workloadIdentity:
    enabled: true
    gsa: "apigee-non-prod@my-hybrid-project.iam.gserviceaccount.com"
```
- If you are using a separate service account for each component (the standard for most prod installations), specify the service account with the component's gsa property. For example:
```
logger:
  gsa: "apigee-logger@my-hybrid-project.iam.gserviceaccount.com"
```
See: gcp.workloadIdentity.enabled, gcp.federatedWorkloadIdentity.enabled, Enabling Workload Identity on GKE or Enabling Workload Identity Federation on AKS and EKS.

Troubleshooting

There is a known issue with the Helm migration tool in the hybrid v1.12 release. Until the issue is resolved, Cassandra backup and restore requires additional steps.

You can follow these steps:

Before or after running the migration tool
Before installing Helm charts

To install the patch for the workaround:

Make sure that your current kubeconfig is pointing to the cluster you want to migrate. You can can perform these steps from any directory.

Create a file named migration-operator-patch.yaml with the following contents:

# migration-operator-patch.yaml
metadata:
  annotations:
    meta.helm.sh/release-name: operator
    meta.helm.sh/release-namespace: apigee-system
  labels:
    app.kubernetes.io/managed-by: Helm

Create a file called migration-datastore-patch.yaml with the following contents:

# migration-datastore-patch.yaml
metadata:
  annotations:
    meta.helm.sh/release-name: datastore
    meta.helm.sh/release-namespace: apigee
  labels:
    app.kubernetes.io/managed-by: Helm

Run the following kubectl commands:

kubectl patch clusterrole apigee-cassandra-backup --patch-file ./migration-operator-patch.yaml
kubectl patch clusterrole apigee-cassandra-restore --patch-file ./migration-operator-patch.yaml
kubectl patch clusterrolebinding apigee-cassandra-backup --patch-file ./migration-operator-patch.yaml
kubectl patch clusterrolebinding apigee-cassandra-restore --patch-file ./migration-operator-patch.yaml
kubectl patch -n apigee cronjob apigee-cassandra-backup --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee certificate apigee-cassandra-backup-tls --patch-file ./migration-datastore-patch.yaml --type merge
kubectl patch -n apigee secret apigee-cassandra-backup-svc-account --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee secret apigee-cassandra-backup-key-file --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee ServiceAccount apigee-cassandra-backup-sa --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee job apigee-cassandra-restore --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee certificate apigee-cassandra-restore-tls --patch-file ./migration-datastore-patch.yaml --type merge
kubectl patch -n apigee secret apigee-cassandra-restore-svc-account --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee secret apigee-cassandra-restore-key-file --patch-file ./migration-datastore-patch.yaml
kubectl patch -n apigee ServiceAccount apigee-cassandra-restore-sa --patch-file ./migration-datastore-patch.yaml

Note: You can ignore any not found errors.

Clean up the patch files using the following commands:

rm migration-operator-patch.yaml
rm migration-datastore-patch.yaml

Next step

Continue your installation of the Apigee hybrid Helm charts with the instructions in Installing and managing Apigee hybrid with Helm charts.

Important: When following the instructions for your migrated cluster, you must omit the ‑‑atomic flag from any Helm commands to avoid accidentally deleting resources if the helm upgrade command fails.

Output of -help

./apigee-helm-migration --help

Usage of ./apigee-helm-migration:
  -apigee-namespace string
      namespace used for apigee resources (default "apigee")
  -components string
      CSV of components to migrate. If empty then all components are migrated. Valid values are: operator,datastore,redis,ingress-manager,telemetry,orgs,envs,env-groups
  -dry-run
      perform a dry-run
  -env string
      restrict migration to a singular supplied env. If empty then all envs detected in the cluster are migrated
  -env-group string
      restrict migration to a singular supplied envGroup. If empty then all envGroups detected in the cluster are migrated
  -kubeconfig string
      (optional) absolute path to the kubeconfig file (default "/usr/local/google/home/example/.kube/config")
  -org string
      restrict migration to a singular supplied org. If empty then all orgs detected in the cluster are migrated
  -v	Increased logging verbosity
  -y	don't prompt for confirmation or for configuration of Helm releases