Migrating containers to Google Cloud: Migrate OpenShift projects to GKE Enterprise

Last reviewed 2022-01-24 UTC

This document helps you to plan, design, and implement migration of your projects from OpenShift to GKE Enterprise. If done incorrectly, moving your workloads from one environment to another can be a challenging task, so plan and execute your migration carefully.

This document is part of a multi-part series about migrating to Google Cloud. If you're interested in an overview of the series, see Migration to Google Cloud: Choosing your migration path.

This document is part of a series that discusses migrating containers to Google Cloud:

This document is useful if you're planning to migrate OpenShift projects to GKE Enterprise. This document is also useful if you're evaluating the opportunity to migrate and want to explore what it might look like.

This document relies on concepts that are discussed in Migration to Google Cloud: Getting started, in Migrating containers to Google Cloud: Migrating Kubernetes to GKE, in Migrating containers to Google Cloud: Migrating from OpenShift to GKE Enterprise, and in Best practices for GKE networking. This document links to the preceding documents where appropriate.

The guidance in this document assumes that you want to execute a lift and shift migration of your workloads. In a lift and shift migration, you apply only the minimum changes that you need for your workloads to operate in the target GKE Enterprise environment.

To migrate OpenShift resources to GKE Enterprise, you map and convert them to their Kubernetes equivalents. This document describes the migration of the following OpenShift project configuration resources necessary to deploy and operate your workloads to GKE Enterprise:

To migrate OpenShift project configuration and related resources to GKE Enterprise, we recommend that you do the following:

  1. Export OpenShift project configuration resource descriptors.
  2. Map the OpenShift project configuration resources to Kubernetes resources.
  3. Create Kubernetes resources that map to OpenShift project configuration resources.
  4. Manage the Kubernetes resources using Config Sync.

This document provides examples of how you can complete the migration steps.

Export OpenShift project configuration resource descriptors

To export the OpenShift project configuration resources, we recommend that you do the following:

  1. Export OpenShift project descriptors.
  2. Export cluster-scoped resource descriptors.
  3. Export project-scoped resource descriptors.

The descriptors that you export from an OpenShift cluster include fields that describe the configuration and the status of resources, such as the spec and status fields. The descriptors also include fields that hold resource status information, such as the metadata.managedFields field. Kubernetes and OpenShift manage the fields that hold resource status information and their values for you. To simplify the assessment of OpenShift resource descriptors, we recommend that you do the following for each resource descriptor:

  1. Record the fields that hold dynamically generated resource status information along with their values, such as the following:

    • Any field nested under metadata.annotations that starts with the openshift.io prefix
    • metadata.creationTimestamp
    • metadata.generation
    • metadata.managedFields
    • metadata.resourceVersion
    • metadata.selfLink
    • metadata.uid
    • status
  2. Remove the fields that hold dynamically generated resource status information from the resource descriptor.

To export OpenShift project configuration resource descriptors, you use the OpenShift command-line interface (oc CLI). To export resource descriptors in oc CLI, you need to authenticate with the cluster-admin role. For a list of all the OpenShift resources that the oc CLI supports, run the oc api-resources command.

Export OpenShift project descriptors

This section describes how to export project descriptors. We recommend that you exclude OpenShift projects that run system components, such as the istio-system component, and exclude OpenShift projects that have names starting with openshift-, kube-, or knative-. OpenShift manages these OpenShift projects for you, and they're out of the scope of this migration because you don't use them to deploy your workloads. To export OpenShift project descriptors, do the following for each OpenShift cluster:

  1. In a terminal that has access to the OpenShift cluster, get the list of OpenShift projects by using the oc get command:

    oc get projects
    

    The output is similar to the following:

    NAME                 DISPLAY NAME      STATUS
    example-project                        Active
    ...
    

    The output displays a list of the OpenShift projects that are currently set up in your OpenShift cluster.

  2. For each OpenShift project in the list, export its descriptor in YAML file format, display the output, and save it to a file using the tee command, for later processing. For example, export the descriptor of an example-project OpenShift project:

    oc get project example-project -o yaml | tee project-example-project.yaml
    

    The output is similar to the following:

    apiVersion: project.openshift.io/v1
    kind: Project
    metadata:
      annotations:
      name: example-project
    spec:
      finalizers:
      - kubernetes
    

    The output displays the descriptor of the example-project OpenShift project in YAML file format. The output is saved to the project-example-project.yaml file.

Export cluster-scoped resource descriptors

This section describes how to export the descriptors for resources that have a cluster scope, not including security context constraints. For information about migrating security policies, see Migrating from OpenShift to GKE Enterprise: Migrate OpenShift SCCs to Policy Controller Constraints. To export other resource descriptors, do the following for each OpenShift cluster:

  1. In your terminal, get the list of ClusterResourceQuotas:

    oc get clusterresourcequotas
    

    The output is similar to the following:

    NAME       AGE
    for-name   6m15s
    for-user   4s
    ...
    

    The output displays a list of ClusterResourceQuotas that are currently set up in your OpenShift cluster.

  2. For each ClusterResourceQuota in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the for-name ClusterResourceQuota:

    oc get clusterresourcequota for-name -o yaml | tee clusterresourcequota-for-name.yaml
    

    The output is similar to the following:

    apiVersion: quota.openshift.io/v1
    kind: ClusterResourceQuota
    metadata:
      name: for-name
    spec:
      quota:
        hard:
          pods: "10"
          secrets: "20"
      selector:
        annotations: null
        labels:
          matchLabels:
            name: frontend
    

    The output displays the descriptor of the for-name ClusterResourceQuota in YAML format. The output is saved to the clusterresourcequota-for-name.yaml file.

  3. Get the list of ClusterRoles:

    oc get clusterroles
    

    The output is similar to the following:

    NAME                           CREATED AT
    admin                          2021-02-02T06:17:02Z
    aggregate-olm-edit             2021-02-02T06:17:59Z
    aggregate-olm-view             2021-02-02T06:18:01Z
    alertmanager-main              2021-02-02T06:48:26Z
    basic-user                     2021-02-02T06:26:42Z
    ...
    

    The output displays a list of ClusterRoles that are currently set up in your OpenShift cluster. The list of ClusterRoles includes OpenShift default ClusterRoles, and ClusterRoles that refer to OpenShift system components. We recommend that you assess all the ClusterRoles in the list to evaluate which roles you need to migrate, and which roles aren't applicable in the target GKE Enterprise environment.

  4. For each ClusterRole in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the admin ClusterRole:

    oc get clusterrole admin -o yaml | tee clusterrole-admin.yaml
    

    The output is similar to the following:

    aggregationRule:
      clusterRoleSelectors:
      - matchLabels:
          rbac.authorization.k8s.io/aggregate-to-admin: "true"
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      labels:
        kubernetes.io/bootstrapping: rbac-defaults
      name: admin
    rules:
    - apiGroups:
      - operators.coreos.com
      resources:
      - subscriptions
      verbs:
      - create
      - update
      - patch
      - delete
    ...
    

    The output displays the descriptor of the admin ClusterRole in YAML format. The output is saved to the clusterrole-admin.yaml file.

  5. Get the list of ClusterRoleBindings:

    oc get clusterrolebindings
    

    The output is similar to the following:

    NAME                                        ROLE                                             AGE
    alertmanager-main                           ClusterRole/alertmanager-main                    21d
    basic-users                                 ClusterRole/basic-user                           21d
    cloud-credential-operator-rolebinding       ClusterRole/cloud-credential-operator-role       21d
    cluster-admin                               ClusterRole/cluster-admin                        21d
    cluster-admins                              ClusterRole/cluster-admin                        21d
    cluster-autoscaler                          ClusterRole/cluster-autoscaler                   21d
    cluster-autoscaler-operator                 ClusterRole/cluster-autoscaler-operator          21d
    cluster-monitoring-operator                 ClusterRole/cluster-monitoring-operator          21d
    ...
    

    The output displays a list of ClusterRoleBindings that are currently set up in your OpenShift cluster. The list of ClusterRoleBindings includes ClusterRoleBindings that refer to OpenShift system components. We recommend that you assess all the ClusterRoleBindings in the list to evaluate which bindings you need to migrate, and which bindings aren't applicable in the target GKE Enterprise environment.

  6. For each ClusterRoleBinding in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the cluster-admin ClusterRoleBinding:

    oc get clusterrolebinding cluster-admin -o yaml | tee clusterrolebinding-cluster-admin.yaml
    

    The output is similar to the following:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      labels:
        kubernetes.io/bootstrapping: rbac-defaults
      name: cluster-admin
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - apiGroup: rbac.authorization.k8s.io
      kind: Group
      name: system:masters
    

    The output displays the descriptor of the cluster-admin ClusterRoleBinding in YAML format. The output is saved to the clusterrolebinding-cluster-admin.yaml file.

Export customized NetNamespaces

This section describes how to assess the configuration of multi-tenant isolation. This section applies if you created customized NetNamespaces for any OpenShift project in your cluster to isolate or join network namespaces. If you didn't create customized NetNamespaces, skip to Export project-scoped resource descriptors.

OpenShift automatically creates and manages NetNamespaces for managed OpenShift projects. NetNamespaces for OpenShift-managed projects are out of the scope of this migration.

To export customized NetNamespaces, do the following:

  1. Get the list of NetNamespaces:

    oc get netnamespaces
    

    The output is similar to the following.

    NAME                                          NETID      EGRESS IPS
    default                                       0
    kube-node-lease                               13240579
    kube-public                                   15463168
    kube-system                                   16247265
    openshift                                     9631477
    openshift-apiserver                           12186643
    openshift-apiserver-operator                  6097417
    openshift-authentication                      2862939
    openshift-authentication-operator             723750
    openshift-cloud-credential-operator           11544971
    openshift-cluster-csi-drivers                 7650297
    openshift-cluster-machine-approver            7836750
    openshift-cluster-node-tuning-operator        7531826
    ...
    

    The output displays a list of NetNamespaces that are currently set up in your OpenShift cluster.

  2. For each NetNamespace in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the default NetNamespace:

    oc get netnamespace example-project -o yaml | tee netnamespace-example-project.yaml
    

    For NetNamespaces that don't have the same netid value, the output is similar to the following:

    apiVersion: network.openshift.io/v1
    kind: NetNamespace
    metadata:
      name: example-project
    netid: 1234
    netname: example-project
    

    The output displays the descriptor of the example-project NetNamespace in YAML file format. The output is saved to the netnamespace-example-project.yaml file.

    For NetNamespaces that have the same netid value, the output is similar to the following:

    apiVersion: network.openshift.io/v1
    kind: NetNamespace
    metadata:
      name: example-project
    netid: 1234
    netname: example-project
    
    apiVersion: network.openshift.io/v1
    kind: NetNamespace
    metadata:
      name: example-project-2
    netid: 1234
    netname: example-project-2
    

Export project-scoped resource descriptors

To export the descriptors for resources that have a project scope, do the following for each OpenShift project.

  1. In your terminal, select the OpenShift project that you want to assess. For example, select the example-project OpenShift project:

    oc project example-project
    
  2. Get the list of ResourceQuotas:

    oc get resourcequotas
    

    The output is similar to the following:

    NAME        AGE   REQUEST                        LIMIT
    gpu-quota   6s    requests.nvidia.com/gpu: 1/1
    ...
    

    The output displays a list of ResourceQuotas that are currently set up in your OpenShift cluster for the selected OpenShift project.

  3. For each ResourceQuota in the list, export its descriptor in YAML format, display the output, and save it to a file for later processing. For example, export the descriptor of the gpu-quota ResourceQuota:

    oc get resourcequota gpu-quota -o yaml | tee resourcequota-gpu-quota.yaml
    

    The output is similar to the following:

    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: gpu-quota
      namespace: example-project
    spec:
      hard:
        requests.nvidia.com/gpu: "1"
    

    The output displays the descriptor of the gpu-quota ResourceQuota in YAML file format. The output is saved to the resourcequota-gpu-quota.yaml file.

  4. Get the list of Roles:

    oc get roles
    

    The output is similar to the following:

    NAME             CREATED AT
    example          2021-02-02T06:48:27Z
    
    ...
    

    The output displays a list of Roles that are currently set up in your OpenShift cluster for the selected OpenShift project. The list of Roles includes Roles that refer to OpenShift system components. We recommend that you assess all the Roles in the list to evaluate which roles you need to migrate, and which roles aren't applicable in the target GKE Enterprise environment.

  5. For each Role in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the example Role:

    oc get role example -o yaml | tee role-example.yaml
    

    The output is similar to the following:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: example
      namespace: example-project
    rules:
    - apiGroups:
      - ""
      resources:
      - services
      - endpoints
      - pods
      verbs:
      - get
      - list
      - watch
    

    The output displays the descriptor of the example Role in YAML file format. The output is saved to the role-example.yaml file.

  6. Get the list of RoleBindings:

    oc get rolebindings
    

    The output is similar to the following:

    NAME                               ROLE                                           AGE
    machine-config-controller-events   ClusterRole/machine-config-controller-events   21d
    machine-config-daemon-events       ClusterRole/machine-config-daemon-events       21d
    example                            Role/example                                   21d
    system:deployers                   ClusterRole/system:deployer                    21d
    system:image-builders              ClusterRole/system:image-builder               21d
    system:image-pullers               ClusterRole/system:image-puller                21d
    ...
    

    The output displays a list of RoleBindings that are set up in your OpenShift cluster for the selected OpenShift project. The list of RoleBindings includes RoleBindings that refer to OpenShift system components. We recommend that you assess all the RoleBindings in the list to evaluate which bindings you need to migrate, and which bindings aren't applicable in the target GKE Enterprise environment.

  7. For each RoleBinding in the list, export its descriptor in YAML file format, display the output, and save it to a file for later processing. For example, export the descriptor of the example RoleBinding:

    oc get rolebinding example -o yaml | tee rolebinding-example.yaml
    

    The output is similar to the following:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: example
      namespace: example-project
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: example
    subjects:
    - kind: ServiceAccount
      name: example
      namespace: example-ns
    

    The output displays the descriptor of the example RoleBinding in YAML file format. The output is saved to the rolebinding-example.yaml file.

  8. Get the list of EgressNetworkPolicies:

    oc get egressnetworkpolicies
    

    The output is similar to the following:

    NAME      AGE
    default   2m2s
    ...
    

    The output displays a list of EgressNetworkPolicies that are currently set up in your OpenShift cluster for the selected OpenShift project.

  9. For each EgressNetworkPolicy in the list, export its descriptor in YAML file format, display the output, and save it to a file for l