Managing cloud infrastructure using kpt

Last reviewed 2023-03-15 UTC

This tutorial introduces kpt, an open source tool by Google that lets you work with Kubernetes configurations (also known as manifests): package them, pull them, update them, and modify them. kpt is an alternative to template-based tools when you want to keep a clean separation between configurations and operations on those configurations. kpt lets you reuse and share code that is acting on the configurations (either to modify or inspect them).

This tutorial also showcases how you can combine kpt with other Google solutions like Config Sync and GKE Enterprise security blueprints. Whether you are a developer working with Kubernetes or a platform engineer managing a Kubernetes-based platform, this tutorial lets you discover how you can use kpt in your own Kubernetes-related workflows. This tutorial assumes that you are familiar with Kubernetes and Google Cloud.

Declarative configuration for cloud infrastructure is a well-established practice in the IT industry. It brings a powerful abstraction of the underlying systems. This abstraction frees you from having to manage low-level configuration details and dependencies. Therefore, declarative configuration has an advantage compared to imperative approaches, such as operations performed in graphical and command line interfaces.

The Kubernetes resource model has been influential in making declarative configuration approaches mainstream. Because the Kubernetes API is fully declarative by nature, you only tell Kubernetes what you want, not how to achieve what you want. The Kubernetes API lets you cleanly separate the configuration (whether desired or real) from operations on the configuration (adding, removing, and modifying objects). In other words, in the Kubernetes resource model, the configuration is data, and not code.

This separation of configuration from operations has many advantages: people and automated systems can understand and work on the configuration, and the software modifying the configuration is easily reusable. This separation also lets you easily implement a GitOps methodology (as defined in the GitOps-style continuous delivery with Cloud Build tutorial).

In this tutorial, you explore this separation of configuration declaration from configuration operations using kpt. This tutorial highlights the following features of kpt:

  • Package management: download and update Kubernetes configuration packages.
  • Functions: run arbitrary pieces of code to either modify or validate your configurations.
  • Function pipeline: a set of functions that the package author has included with the package.
  • Resource management: apply, update, and delete the resources that correspond to your configurations in a Kubernetes cluster.

Objectives

  • Create a Google Kubernetes Engine (GKE) cluster.
  • Use kpt to download an existing set of Kubernetes configurations.
  • Use kpt functions to customize the configurations.
  • Apply your configuration to the GKE cluster.
  • Use kpt to pull in some upstream changes for your configuration.
  • Use kpt in a real-world scenario to harden your GKE cluster.

Costs

In this document, you use the following billable components of Google Cloud:

  • Google Kubernetes Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  5. Make sure that billing is enabled for your Google Cloud project.

  6. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  7. Configure Cloud Shell to use your project:

    gcloud config set project PROJECT_ID
    
  8. In Cloud Shell, enable the Google Kubernetes Engine and Cloud Source Repositories APIs:

    gcloud services enable container.googleapis.com \
       sourcerepo.googleapis.com
    

When you finish this tutorial, you can avoid continued billing by deleting the resources that you created. For more details, see Cleaning up.

Creating a GKE cluster

In this section, you create the GKE cluster where you deploy configurations later in the tutorial.

  1. In Cloud Shell, create a GKE cluster:

    gcloud container clusters create kpt-tutorial \
       --num-nodes=1 --machine-type=n1-standard-4 \
       --zone=us-central1-a --enable-network-policy
    
  2. Verify that you have access to the cluster. The following command returns information about the node or nodes that are in the cluster.

    kubectl get nodes
    

Applying a kpt package

In this section, you use kpt to download a set of configurations, customize them, and apply them to the cluster that you created in the previous section. kpt should be installed inside your Cloud Shell environment. If it's not, install it with the following commands:

  1. In Cloud Shell, install kpt:

    sudo apt update && sudo apt-get install google-cloud-sdk-kpt
    
  2. Download an example set of configurations. For more information, see kpt pkg get.

    kpt pkg get https://github.com/GoogleContainerTools/kpt.git/package-examples/wordpress@v0.9
    

    The preceding command downloads the wordpress sample package that is available in the kpt GitHub repository, at the version tagged v0.9.

  3. Examine the package contents: kpt pkg tree.

    kpt pkg tree wordpress
    

    The output looks like the following:

    Package "wordpress"
    ├── [Kptfile]  Kptfile wordpress
    ├── [service.yaml]  Service wordpress
    ├── deployment
    │   ├── [deployment.yaml]  Deployment wordpress
    │   └── [volume.yaml]  PersistentVolumeClaim wp-pv-claim
    └── Package "mysql"
        ├── [Kptfile]  Kptfile mysql
        ├── [deployment.yaml]  PersistentVolumeClaim mysql-pv-claim
        ├── [deployment.yaml]  Deployment wordpress-mysql
        └── [deployment.yaml]  Service wordpress-mysql
    

    The package contains two packages the top level one wordpress and a subpackage wordpress/mysql, both of these packages contain a metadata file Kptfile. Kptfile is only consumed by kpt itself and has data about the upstream source, customization and validation of the package

  4. Update the label of the package

    The author of the package added a rendering pipeline which is often used to convey the expected customizations.

    less wordpress/Kptfile
    

    The contents should look something like this:

    apiVersion: kpt.dev/v1
    kind: Kptfile
    metadata:
      name: wordpress
    upstream:
      type: git
      git:
        repo: https://github.com/GoogleContainerTools/kpt
        directory: /package-examples/wordpress
        ref: v0.9
      updateStrategy: resource-merge
    upstreamLock:
      type: git
      git:
        repo: https://github.com/GoogleContainerTools/kpt
        directory: /package-examples/wordpress
        ref: package-examples/wordpress/v0.9
        commit: b9ea0bca019dafa9f9f91fd428385597c708518c
    info:
      emails:
        - kpt-team@google.com
      description: This is an example wordpress package with mysql subpackage.
    pipeline:
      mutators:
        - image: gcr.io/kpt-fn/set-labels:v0.1
          configMap:
            app: wordpress
      validators:
        - image: gcr.io/kpt-fn/kubeval:v0.3
    

    You can use your favorite editor to change the parameters of the set-label function from app: wordpress to app: my-wordpress

  5. Edit the MySQL deployment wordpress/mysql/deployment.yaml using your favorite code editor to change the MySQL version. Also, to further strengthen the security, change the MYSQL_ROOT_PASSWORD variable to MYSQL_PASSWORD, and add the following variables:

    • MYSQL_USER
    • MYSQL_RANDOM_ROOT_PASSWORD
    • MYSQL_DATABASE

    Before:

    [...]
      containers:
        - name: mysql
          image: mysql:5.6
          ports:
            - name: mysql
              protocol: TCP
              containerPort: 3306
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
    [...]
    

    After:

    [...]
      containers:
        - name: mysql
          image: mysql:8.0
          ports:
            - name: mysql
              protocol: TCP
              containerPort: 3306
          env:
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            - name: MYSQL_RANDOM_ROOT_PASSWORD
              value: '1'
            - name: MYSQL_USER
              value: wordpress
            - name: MYSQL_DATABASE
              value: wordpress
    [...]
    
  6. Edit the Wordpress deployment wordpress/deployment/deployment.yaml using your favorite code editor to change the Wordpress version and add a WORDPRESS_DB_USER variable.

    Before:

    [...]
      containers:
        - name: wordpress
          image: wordpress:6.1-apache
          ports:
            - name: wordpress
              protocol: TCP
              containerPort: 80
          env:
            - name: WORDPRESS_DB_HOST
              value: wordpress-mysql
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
    [...]
    

    After:

    [...]
      containers:
        - name: wordpress
          image: wordpress:4.8-apache
          ports:
            - name: wordpress
              protocol: TCP
              containerPort: 80
          env:
            - name: WORDPRESS_DB_HOST
              value: wordpress-mysql
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            - name: WORDPRESS_DB_USER
              value: wordpress
    [...]
    

    Unlike tools that operate only through parameters, kpt allows you to edit files in place using an editor and then will still merge the upstream updates. You can edit the deployment.yaml directly without having to craft a patch or creating a function in the pipeline.

  7. Annotate the configuration with sample-annotation: sample-value.

    kpt fn eval wordpress --image gcr.io/kpt-fn/set-annotations:v0.1 \
      -- sample-annotation=sample-value
    

    The output should look something like this:

    [RUNNING] "gcr.io/kpt-fn/set-annotations:v0.1"
    [PASS] "gcr.io/kpt-fn/set-annotations:v0.1"
    

    To see the new annotation you can examine any configuration value, a simple one to see is wordpress/service.yaml

    In this example we did a customization using a function in a way that the package author didn't plan for. kpt is able to support declarative and imperative function execution to allow for a wide range of scenarios.

    If this package was developed using infrastructure-as-code we would need to go to the source of the package and edit code there.

  8. Run the pipeline and validate the changes using kubeval through kpt.

    kpt fn render wordpress
    

    The package author has included a validation step in the pipeline:

    ...
    validators:
        - image: gcr.io/kpt-fn/kubeval:v0.3
    

    The successful output of this rendering pipeline looks like this:

    Package "wordpress/mysql":
    [RUNNING] "gcr.io/kpt-fn/set-labels:v0.1"
    [PASS] "gcr.io/kpt-fn/set-labels:v0.1" in 1.3s
    
    Package "wordpress":
    [RUNNING] "gcr.io/kpt-fn/set-labels:v0.1"
    [PASS] "gcr.io/kpt-fn/set-labels:v0.1" in 1.3s
    [RUNNING] "gcr.io/kpt-fn/kubeval:v0.3"
    [PASS] "gcr.io/kpt-fn/kubeval:v0.3" in 3.7s
    
    Successfully executed 3 function(s) in 2 package(s).
    

    This step is an example of the benefits of the Kubernetes resource model: because your configurations are represented in this well-known model, you can use existing Kubernetes tools like kubeval.

  9. Examine the differences between your local version of the configurations and the upstream configuration:

    kpt pkg diff wordpress
    
  10. Initialize the package for deployment:

    kpt live init wordpress
    

    The preceding command builds an inventory of the configurations that are in the package. Among other things, kpt uses the inventory to prune configurations when you remove them from the package. For more information, see kpt live.

  11. Create a secret that contains the MySQL password:

    kubectl create secret generic mysql-pass --from-literal=password=foobar
    
  12. Apply the configurations to your GKE cluster:

    kpt live apply wordpress
    

    The output looks like the following:

    installing inventory ResourceGroup CRD.
    inventory update started
    inventory update finished
    apply phase started
    service/wordpress apply successful
    service/wordpress-mysql apply successful
    deployment.apps/wordpress apply successful
    deployment.apps/wordpress-mysql apply successful
    persistentvolumeclaim/mysql-pv-claim apply successful
    persistentvolumeclaim/wp-pv-claim apply successful
    apply phase finished
    reconcile phase started
    service/wordpress reconcile successful
    service/wordpress-mysql reconcile successful
    deployment.apps/wordpress reconcile pending
    deployment.apps/wordpress-mysql reconcile pending
    persistentvolumeclaim/mysql-pv-claim reconcile pending
    persistentvolumeclaim/wp-pv-claim reconcile pending
    persistentvolumeclaim/wp-pv-claim reconcile successful
    persistentvolumeclaim/mysql-pv-claim reconcile successful
    deployment.apps/wordpress-mysql reconcile successful
    deployment.apps/wordpress reconcile successful
    reconcile phase finished
    inventory update started
    inventory update finished
    apply result: 6 attempted, 6 successful, 0 skipped, 0 failed
    reconcile result: 6 attempted, 6 successful, 0 skipped, 0 failed, 0 timed out
    
  13. Wait a few minutes, and then verify that everything is running as expected:

    kpt live status wordpress
    

    The output looks like the following:

    inventory-88521939/deployment.apps/default/wordpress is Current: Deployment is available. Replicas: 1
    inventory-88521939/persistentvolumeclaim/default/wp-pv-claim is Current: PVC is Bound
    inventory-88521939/service/default/wordpress-mysql is Current: Service is ready
    inventory-88521939/persistentvolumeclaim/default/mysql-pv-claim is Current: PVC is Bound
    inventory-88521939/deployment.apps/default/wordpress-mysql is Current: Deployment is available. Replicas: 1
    inventory-88521939/service/default/wordpress is Current: Service is ready
    

Updating your local package

In this section, you update your local version of the package with some changes from the upstream package.

  1. Create a Git repository for your configurations and configure your email and name. You need a local Git repository to be able to update the package. Replace YOUR_EMAIL with your email address and replace YOUR_NAME with your name.

    cd wordpress/
    git init -b main
    git config user.email "YOUR_EMAIL"
    git config user.name "YOUR_NAME"
    
  2. Commit your configurations:

    git add .
    git commit -m "First version of Wordpress package"
    cd ..
    
  3. Update your local package. In this step, you pull the v0.10 version from upstream.

    kpt pkg update wordpress@v0.10
    
  4. Observe that updates from upstream are applied to your local package:

    cd wordpress/
    git diff
    

    In this case, the update has changed the wordpress deployment volume from 3Gi to 4Gi. You can also see you own changes.

  5. Commit your changes:

    git commit -am "Update to package version v0.10"
    
  6. Apply the new version of the package:

    kpt live apply .
    

Removing a resource and a package

Because kpt tracks the resources that it creates, it can prune resources from the cluster when you delete resources from the package. It can also completely remove a package from the cluster. In this section, you remove a resource from the package, and then remove the package.

  1. Remove the service.yaml file from the package:

    git rm service.yaml
    git commit -m "Remove service"
    
  2. Apply the change, and then verify that kpt pruned the wordpress service:

    kpt live apply .
    kubectl get svc
    
  3. Remove the rest of the package from the cluster, and then verify that you have nothing left in the cluster:

    kpt live destroy .
    kubectl get deployment
    

Using kpt to harden your GKE

The kpt live command is not the only way that you can apply a package to a Kubernetes cluster. In this section, you use kpt with Config Sync in a basic but realistic scenario. The Config Sync tool lets you manage your configuration centrally, uniformly, and declaratively for all your Kubernetes clusters from a Git repository. GKE Enterprise (which you use in this tutorial) includes Config Sync.

The GKE Enterprise security blueprints provide you with a range of prepackaged security settings for your GKE Enterprise-based workloads. In this section, you use the restricting traffic blueprint (and its directory in the GitHub repository). You use kpt to download the default-deny package. The package uses Kubernetes network policies to deny any traffic in your GKE cluster by default (except for DNS resolution). To apply the configurations, you then commit the configurations to the Config Sync Git repository.

Install Config Sync

In this section, you create the Git repository that Config Sync needs, and then you install Config Sync on your GKE cluster.

  1. In Cloud Shell, use Cloud Source Repositories to create a Git repository for Config Sync:

    cd ~
    gcloud source repos create config-management
    
  2. Generate an SSH key pair to authenticate against the Git reposit