Upgrade guide

This guide provides general guidance on how to upgrade and existing Manufacturing Data Engine (MDE) installation using Terraform.

If you have modified Terraform scripts you might need additional steps. This is a generic upgrade guide so always make sure that you also read the full releases notes for the specific version you are deploying as this may include information you need to take into consideration.

Before you begin

Some Dataflow jobs need to be stopped manually before upgrading. The release notes list the jobs you have to stop manually for the specific version you are upgrading to.

This guide requires the follow prerequisites:

1. You are using the default deployment package.

2. Your client environment has the required CLI tools installed in their most updated version:

   • Google Cloud CLI with the following additional components installed:

     • kubectl
     • cbt
     • Terraform CLI ( 1.9.x or later)
     • Helm CLI (3.9.x or later)

   You can use any client environment to deploy MDE, but you can save time by deploying from Cloud Shell since it has most of the required tools already installed.

   Activate Cloud Shell

3. You have full access to the MDE Google Cloud project and access to the configuration files used for the original deployment:

   • Service Account JSON key file: mde-imgs-service-account-key.json.
   • Terraform: input.tfvars.
   • Terraform: backend.conf.

   All gcloud commands in this guide assume that the default project is set to the MDE deployment project. You can set the default project using the following command:

   gcloud config set project PROJECT_ID
   

   Replace PROJECT_ID with the MDE deployment project ID.

4. You have acquired the credentials for the MDE GKE cluster. Use the following command if you haven't already:

   export CLUSTER_NAME="mde-gke"
   export CLUSTER_LOCATION=$(gcloud container clusters list --filter="name:${CLUSTER_NAME}" --format="value(LOCATION)" )
   
   export KUBE_CONFIG_PATH=~/.kube/config
   gcloud container clusters get-credentials ${CLUSTER_NAME} --region ${CLUSTER_LOCATION}
   

Upgrade

This section provides guidance on how perform the upgrade using Terraform.

1. Update the Terraform service account permissions to add the newly required permissions from MDE 1.4.0 and up. They can be added by using the following commands:

   export PROJECT_ID=$(gcloud config get-value project)
   export SA_TERRAFORM="mde-tf"
   
   gcloud projects add-iam-policy-binding ${PROJECT_ID} \
   --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \
   --role='roles/file.editor'
   

2. Backup the old deployment folder using the following command:

   cp -r MDE_FOLDER/MDE_FOLDER_BACKUP
   

3. Download the latest MDE solution release package and extract it:

   1. Download the solution package.
   2. Extract the package to your client environment.
   3. Move into the new release folder using the cd command.

4. Copy the following files from the backup to the new folder using the following commands (files paths may differ):

   cp ../MDE_FOLDER_BACKUP/mde-imgs-service-account-key.json .
   cd deployment/terraform
   cp ../../../MDE_FOLDER_BACKUP/deployment/terraform/input.tfvars .
   cp ../../../MDE_FOLDER_BACKUP/deployment/terraform/backend.conf .
   

5. If you are upgrading from v.1.4.x to v1.5.x, run the pre-upgrade script included in the release package:

   # Execute from upgrade/1.5.0 directory
   sh migrate-metadata-instance-bq-table.sh
   

6. Reload the Terraform state using the following command:

   # Execute from deployment/terraform directory
   terraform init -backend-config=backend.conf -reconfigure
   

7. Create a Terraform plan.

   Once the input parameters are ready, you must create a Terraform plan using the following command. You can use the plan to verify what artifacts and configurations will be created in the project.

   terraform plan -var-file=./input.tfvars -out=./tfplan
   

8. Apply the Terraform plan using the following command:

   terraform apply ./tfplan
   

9. Verify that the deployment succeeded.

   After the terraform apply command finishes executing, you should see a success message that looks similar to the following (the actual number will depend on the specific deployment options and version):

   Apply complete! Resources: 1 added, 34 changed, 0 destroyed.
   

Post Upgrade

After the upgrade is complete, you need to perform one or more of the following steps depending on the status of the earlier MDE deployment.

Natural key hash generation

If you are upgrading from 1.4.x to 1.5.0, you need to follow the steps outlined on this guide before anything else.

BigQuery views for existing types

If you are upgrading from a release before 1.4.x, then the created types won't have an analytics view created for it as this feature was introduced in 1.4.0. You'll need to activate the existing types once again to trigger the analytics view recreation. You can check the /activate endpoint in the Postman kit that's shipped with the release.

GKE cluster filestore driver

If you experience an issue that the GKE cluster pods are starting or stuck in ContainerCreating, then it could be the result of the filestore driver not being enabled. You can enable it by following the steps in the deployment guide