Symptom
After upgrading Apigee hybrid, your Universal Data Collection Agent (UDCA) and Connect Agent pods throw a permission issue error.
Error messages
Your UDCA pod throws this error in the logs:
Permission 'apigee.runtimeconfigs.get' denied on resource 'organizations/org-example/runtimeConfig' (or it may not exist)\\\",\\n \\\"status\\\": \\\"PERMISSION_DENIED\\
In addition, your Management API for Runtime (MART) pod throws this error regarding the Connect Agent:
Permission 'apigeeconnect.endpoints.connect' denied on resource '//apigeeconnect.googleapis.com/projects/org-example/endpoints/APIGEE_MART' (or it may not exist).
Common diagnosis steps
Collect the Apigee hybrid must-gather script from your cluster and the overrides.yaml file for the cluster.
Possible causes
| Cause | Description |
|---|---|
| Incorrect credentials | You might have entered the wrong credentials in the overrides.yaml file. |
| SA files located in wrong directory | Your Service Account (SA) files might be residing outside the $APIGEE_HELM_CHARTS_HOME directory. |
Cause 1: Incorrect credentials
The serviceAccountPath or serviceAccountRef used in your overrides.yaml file for UDCA, MART, and Connect Agent pods are incorrect. This may be because the overrides.yaml file was generated through automation or custom scripts, such as by using Terraform.
Diagnosis
-
Verify the IAM roles and permissions assigned to the service accounts you use.
gcloud iam service-accounts get-iam-policy SERVICE_ACCOUNT_EMAIL
gcloud projects get-iam-policy PROJECT_IDWhere:
- SERVICE_ACCOUNT_EMAIL is the email address of the service account.
- PROJECT_ID is the Google Cloud project associated with the service account.
-
Review your
overrides.yamlfile to make sure the referenced service account matches the one used in the project IAM policy and roles.
Example
An example of a working IAM policy for the service account for a non-prod setup:
gcloud iam service-accounts get-iam-policy apigee-non-prod@org-example.iam.gserviceaccount.com
Sample output:
bindings: - members: - serviceAccount:org-example.svc.id.goog[apigee/apigee-cassandra-backup-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-cassandra-default] - serviceAccount:org-example.svc.id.goog[apigee/apigee-cassandra-guardrails-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-connect-agent-org-example-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-mart-org-example-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-metrics-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-runtime-org-example-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-synchronizer-org-example-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-udca-org-example-sa] - serviceAccount:org-example.svc.id.goog[apigee/apigee-watcher-org-example-sa] role: roles/iam.workloadIdentityUser etag: BwYw33-1N8s= version: 1
An example of validating roles:
gcloud projects get-iam-policy org-example
Sample output:
bindings: - members: - serviceAccount:apigee-non-prod@org-example.iam.gserviceaccount.com role: roles/apigee.analyticsAgent - members: - serviceAccount:apigee-non-prod@org-example.iam.gserviceaccount.com role: roles/apigee.runtimeAgent - members: - serviceAccount:apigee-non-prod@org-example.iam.gserviceaccount.com role: roles/apigee.synchronizerManager - members: - serviceAccount:apigee-non-prod@org-example.iam.gserviceaccount.com role: roles/apigeeconnect.Agent - members: - serviceAccount:apigee-non-prod@org-example.iam.gserviceaccount.com role: roles/storage.objectAdmin
For production, run the same commands against the production service account for apigee-udca and check for the IAM roles that matches the installation section.
Resolution
If the service account file used in your overrides.yaml does not match the correct service account in the project IAM binding and roles, you should delete and recreate the service account, and correct the service account file used in the overrides.yaml.
Cause 2: SA files located in wrong directory
This issue occurs when setting up the helm charts and creating the service account.
Diagnosis
Check the location of your service account credential file to ensure that it resides in the correct helm chart directory. The service account JSON file should reside under the following directories:
$APIGEE_HELM_CHARTS_HOME/apigee-datastore$APIGEE_HELM_CHARTS_HOME/apigee-telemetry$APIGEE_HELM_CHARTS_HOME/apigee-org$APIGEE_HELM_CHARTS_HOME/apigee-env
Example
For example, check the helm chart directories by listing the apigee-datastore directory, and do the same for apigee-org, apigee-env, and apigee-telemetry directories.
Sample Command:
Make sure you are in the correct directory:
pwd
Sample output:
/usr/local/home/example/Hybrid/helm-charts
├── apigee-datastore │ ├── org-example-apigee-non-prod.json │ └── templates ├── apigee-env │ ├── org-example-apigee-non-prod.json │ └── templates ├── apigee-ingress-manager │ └── templates ├── apigee-operator │ ├── etc │ │ ├── crds │ │ │ ├── crd │ │ │ │ ├── bases │ │ │ │ └── patches │ │ │ └── default │ │ ├── examples │ │ │ └── helmfiles │ │ └── tools │ │ └── service-accounts │ │ ├── org-example-apigee-cassandra.json │ │ ├── org-example-apigee-logger.json │ │ ├── org-example-apigee-mart.json │ │ ├── org-example-apigee-metrics.json │ │ ├── org-example-apigee-runtime.json │ │ ├── org-example-apigee-synchronizer.json │ │ ├── org-example-apigee-udca.json │ │ └── org-example-apigee-watcher.json │ └── templates ├── apigee-org │ ├── org-example-apigee-non-prod.json │ └── templates ├── apigee-redis │ └── templates ├── apigee-telemetry │ ├── org-example-apigee-non-prod.json │ └── templates ├── apigee-virtualhost │ ├── certs │ └── templates └── cluster-check 29 directories, 12 files
Resolution
Copy the Service Account file into the four directories (apigee-datastore, apigee-telemetry, apigee-org, and apigee-env) as per the example tree location above, then rerun the helm charts.