Create patch jobs

You can use Patch to apply operating system patches across a group of virtual machine (VM) instances.

To apply patches to your VMs, complete the following steps:

1. Set up your VM.
2. Run a patch job.

Before you begin

• Review OS Config quotas.
• If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:
  

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI, then initialize it by running the following command:

     gcloud init

  2. Set a default region and zone.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI, then initialize it by running the following command:

  gcloud init

  For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Limitations

• You can deploy and run patch jobs only for VMs in a single Google Cloud project. You cannot run patch jobs across Google Cloud projects, even if the VMs are in a Shared VPC. However, you can view the patch compliance data across projects.
• By default, VM Manager does not patch any VMs that are part of a managed instance group (MIG). Patching these VMs are reported as failures in the patch job. You can override this default behavior when you create the patch job. The following limitations apply when you patch VMs that are part of a MIG:
  • When a MIG repairs a VM, it recreates the VM based on the instance template. This might revert the VM to an unpatched state.
  • Patching VMs can cause unexpected results in a MIG that has autoscaling enabled. Autoscaler deletes patched VMs when the load goes down and creates new VMs, without any patches, using MIG's instance template when the load increases. For example, if the average CPU utilization is less than the target utilization you specified for autoscaling, MIG can remove some of the patched VMs while scaling in.

Supported operating systems

For the full list of operating systems and versions that support Patch, see Operating system details.

Set up your VM

To use the Patch feature, complete the following steps:

1. For all VMs, set up VM Manager.
2. For Windows VMs, Google recommends disabling automatic updates on the VMs. This reduces conflicts between the Windows automatic updates and the Patch service.

Permissions

Owners of a Google Cloud project have full access to run and manage patch jobs. For all other users, you need to grant permissions. You can grant one of the following granular roles:

• roles/osconfig.patchJobExecutor: Contains permissions to run, cancel, get, and list patch jobs. It also contains permissions to view instance details for a patch job.
• roles/osconfig.patchJobViewer: Contains permissions for read-only access to get and list patch jobs. It also contains permissions to view instance details for a patch job.

For example, to grant a user access to run patch jobs, use the following command:

gcloud projects add-iam-policy-binding project-id \
    --member user:user-id@gmail.com \
    --role roles/osconfig.patchJobExecutor

Replace the following:

• project-id: The project ID.
• user-id: The user's Google Workspace username.

Run patch jobs

You can run a patch job by using either the Google Cloud console, Google Cloud CLI, or REST.

When you run a patch job, the patching of the VMs starts simultaneously on all instances specified by the instance filter.

After you have started a patch job, you can monitor your patches using the Patch dashboard. It takes approximately 30 minutes after a patch job starts before the data is populated on the dashboard.

gcloud compute os-config patch-jobs execute instance-filter

gcloud compute os-config patch-jobs execute \
    --instance-filter-names="zones/us-east1-b/instances/instance-1" \
    --description "patch for instance-1"

gcloud compute os-config patch-jobs execute \
    --instance-filter-all \
    --duration="1h30m" --reboot-config="DEFAULT" \
    --apt-dist --windows-exclusive-patches=4339284 \
    --yum-minimal --yum-security \
    --async

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter": instance-filter
}

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "description":"patch instance1 in us-east1-b",
  "duration":"5400s",
  "instanceFilter":{
    "instances":[
      "zones/us-east1-b/instances/instance1"
    ]
  }
}

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter":{
    "groupLabels":[
      {
        "labels":{
          "env":"dev",
          "app":"web"
        }
      }
    ],
    "instanceNamePrefixes":[
      "test-"
    ],
    "zones":[
      "asia-east1-b",
      "asia-east1-c"
    ]
  }
}

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
 "duration":"5400s",
 "instanceFilter":{
   "all":true
 },
 "patchConfig":{
   "rebootConfig":"DEFAULT",
   "apt":{
     "type":"DIST"
   },
   "yum":{
     "security":true,
     "minimal":true
   },
   "windowsUpdate":{
     "exclusivePatches":"4339284"
   }
 }
}

Instance filters

You can specify the instances to be included in a patch job by using filters. The following filters are supported for patch jobs:

• Filter by name: Limit patch job to instances with specific names. Instance names must be specified by using the full URI. Supported URI formats include the following:

  • zones/zone/instances/instance-name

  • projects/project-id/zones/zone/instances/instance-name

  • https://www.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/instance-name

• Filter by name prefix: Limit patch job to instances with a specific prefix in their name.

• Filter by zone: Limit patch job to instances in a specific zone.

• Filter by label: Limit patch job to instances with specific labels.

You can also run patch jobs on all instances in a Google Cloud project by setting the all field in instanceFilter to true. For more information, see example instance filters.

Example instance filters

--instance-filter-all

{
  "instanceFilter":{
    "all":"true"
  }
}

--instance-filter-names="zones/us-east1-b/instances/instance1"

{
  "instanceFilter":{
    "instances":[
      "zones/us-east1-b/instances/instance1"
    ]
  }
}

--instance-filter-name-prefixes="app-"

{
  "instanceFilter":{
    "instanceNamePrefixes":[
      "app-"
    ]
  }
}

--instance-filter-zones="us-east1-b","us-east1-c"

{
  "instanceFilter":{
    "zones":[
      "us-east1-b",
      "us-east1-c"
    ]
  }
}

--instance-filter-group-labels="env=dev,app=web"
--instance-filter-group-labels="env=dev,app=worker"

{
  "instanceFilter":{
    "groupLabels":[
      {
        "labels":{
          "env":"dev",
          "app":"web"
        }
      },
      {
        "labels":{
          "env":"dev",
          "app":"worker"
        }
      }
    ]
  }
}

Combining instance filters

Instance filters can also be combined. For example, to run a patch job for instances that have the prefix test-, that are located in the zone us-east1-c, and that have the labels env=dev and app=web, run the following command:

gcloud compute os-config patch-jobs execute \
    --instance-filter-name-prefixes="test-" \
    --instance-filter-zones="us-east1-c" \
    --instance-filter-group-labels="env=prod,app=web"

Patch configuration

When running a patch job, you can specify parameters to control the patches that are applied on the VM. The patch configuration parameters are platform dependent and are often passed through to the underlying system update tools. The actual patches are sourced from the package repositories (Linux) or the Windows Update server (Windows) that is configured on the VM.

You can specify the following patch configurations for your VMs:

• For Windows, you specify the classification of patches to apply (eg. Security and Critical) or target specific KBs to exclude. For more information about patch classification, see the Microsoft support documentation.

• For RHEL, Rocky Linux, and CentOS, the underlying system is yum.

  • For patches that target RHEL and Rocky Linux VMs, you can specify security and minimal packages.
  • For CentOS VMs, there is no security metadata in the CentOS yum repository. Therefore, you need not specify the security option when updating security packages. If you don't specify any package, the patch job updates all packages, including those with security updates.
  • You can also exclude specific packages. For more information, see the yum man pages.

• For Debian & Ubuntu, the underlying system is apt. For patches that target these VMs, you can specify dist-upgrade or a standard upgrade. You can also exclude specific packages. For more information, see either the Debian man pages or Ubuntu man pages.

• For SuSE, the underlying system is zypper, specifically using zypper patches. For patches that target these VMs, you can specify options such as:

  • with update: update all packages not covered by patches
  • with optional: optional patches are treated as needed
  • The categories or severities of patches to apply

  You can also exclude specific patches.

Optionally, for all the supported operating systems, you can select to install approved patches only by specifying these updates. This lets you enter a list of approved packages or patches. When you select these approved patches, only the approved packages or patches are installed. All other patch configuration parameters are skipped during the update.

Examples

gcloud compute os-config patch-jobs execute \
    --instance-filter-zones="northamerica-northeast1-a" \
    --apt-dist \
    --yum-security \
    --yum-minimal \
    --zypper-categories=security \
    --windows-classifications=critical,security \
    --reboot-config=default

gcloud compute os-config patch-jobs execute --help

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute
{
    "instanceFilter":{
        "zones":[
            "northamerica-northeast1-a"
        ]
    },
    "patchConfig":{
        "apt": {
            "type": "dist-upgrade"
        },
        "yum": {
            "security": true,
            "minimal": true
        },
        "zypper": {
            "categories": ["security"]
        },
        "windowsUpdate": {
            "classifications": ["CRITICAL", "SECURITY"]
        },
        "rebootConfig": "DEFAULT"
    }
}

Maintenance window

A maintenance window is the total length of time that you allow a patch job to run. Patch jobs will timeout if they don't complete within the specified maintenance window.

For example, if you set a maintenance window of 60 minutes, then no new patch jobs will be initiated 60 minutes after the start time. Some processes such as downloading a file or rebooting might occur outside of this maintenance window, however no new patch jobs will be initiated.

Reboot options

When running a patch job, you can specify the reboot options for the patch. The following options are available:

• Default: The agent decides if a reboot is necessary by checking well known signals on each OS. Multiple reboots may occur during the patching and may occur before any patches are installed.
• Always: The machine reboots after the update has completed.
• Never: The machine does not reboot after the update has completed. In some cases, this might mean that not all patches are fully applied.

Pre-patch and post-patch scripts

When running a patch job, you can specify scripts to be run as part of the patching process. These scripts are useful for performing tasks such as shutting down an application and performing health checks.

• Pre-patch scripts run before patching starts. If a system reboot is required before patching starts, the pre-patch script runs before the reboot.
• Post-patch scripts run after patching completes. If a system reboot is required as part of the patching, the post-patch script runs after the reboot.

A patch job accepts one pre-patch and one post-patch script for Linux, and one pre-patch and one post-patch script for Windows. Linux and Windows scripts must be provided using the appropriate flags, parameters, or sections when specified from the Google Cloud CLI, REST, or the Google Cloud console respectively. Linux scripts run on Linux VMs only and Windows scripts run on Windows VMs only.

These script files can either be stored on the VM or in a versioned Cloud Storage bucket. If your Cloud Storage object is not publicly readable, ensure that the default Compute Engine service account for the Google Cloud project has the necessary IAM permissions to read Cloud Storage objects. To ensure that you have the correct permissions, check the permission settings on the Cloud Storage object.

If you want to use a Cloud Storage bucket to store your scripts, create a Cloud Storage bucket and upload your scripts to the bucket.

Examples

gcloud compute os-config patch-jobs execute \
    --instance-filter-zones="northamerica-northeast1-a" \
    --async \
    --pre-patch-linux-executable="/tmp/pre_patch_script.sh" \
    --post-patch-linux-executable="gs://my-patch-scripts/linux/post_patch_script#1523477886880" \
    --pre-patch-windows-executable="C:\\Users\\user\\pre-patch-script.cmd" \
    --post-patch-windows-executable="gs://my-patch-scripts/windows/post_patch_script.ps1#135920493447"

gcloud compute os-config patch-jobs execute --help

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter":{
    "zones":[
      "northamerica-northeast1-a"
    ]
  },
  "patchConfig":{
    "preStep":{
      "linuxExecStepConfig":{
        "localPath":"/tmp/pre_patch_script.sh"
      },
      "windowsExecStepConfig":{
        "interpreter":"SHELL",
        "localPath":"C:\\Users\\user\\pre-patch-script.cmd"
      }
    },
    "postStep":{
      "linuxExecStepConfig":{
        "gcsObject":{
          "bucket":"my-patch-scripts",
          "generationNumber":"1523477886880",
          "object":"linux/post_patch_script"
        }
      },
      "windowsExecStepConfig":{
        "gcsObject":{
          "bucket":"my-patch-scripts",
          "generationNumber":"135920493447",
          "object":"windows/post_patch_script.ps1"
        },
        "interpreter":"POWERSHELL"
      }
    }
  }
}

Patch rollout options

You can choose to either patch VMs one zone at a time (zone by zone), or to patch all zones at once (concurrent zones).

Along with making a selection for the zone roll out, you can also specify a zone disruption budget for your VMs.

Zone disruption budget

A disruption budget is the maximum number (or percentage) of VMs per zone to disrupt at any given moment.

What is considered a disrupted VM?

During patching, a VM is considered disrupted from the time the OS Config agent is notified to begin until patching has completed. This disruption time includes the time to complete reboot and any post-patch steps.

A VM is also counted as a part of the disruption budget if it meets any of the following conditions:

• Patching operation fails when applying the patches
• Patching operation fails when running pre or post patch steps
• Patching operation fails to respond with a success notification before timing out

How disruption budgets work

For zone-by-zone rollouts, if the disruption budget in a zone is exceeded, the patch job stops. This happens because continuing to the next zone requires completion of the patch process in the previous zone.

For example, if the disruption budget has a value of 10, and 8 VMs fail to patch in the current zone, the patch job continues to patch 2 VMs at a time until the zone is completed. When that zone is completed successfully, patching begins with 10 VMs at a time in the next zone. If 10 VMs in the next zone fail to patch, the patch job stops.

Examples

gcloud compute os-config patch-jobs execute \
   --instance-filter-all \
   --rollout-mode=zone-by-zone \
   --rollout-disruption-budget=10

gcloud compute os-config patch-jobs execute \
   --instance-filter-all \
   --rollout-mode=concurrent-zones \
   --rollout-disruption-budget-percent=50

POST https://osconfig.googleapis.com/v1/projects/project-id/patchJobs:execute

{
  "instanceFilter":{
    "zones":[
      "us-central1-a",
      "us-central1-c",
      "us-central1-f"
    ]
  },
  "rollout": {
    "disruptionBudget": {
      "percent": 25
    },
    "mode": "CONCURRENT_ZONES"
  }
}

Enable patching Microsoft software on Windows VMs

When you run a patch job on Windows VMs, by default, Patch applies only the patches for the Windows operating system.

You can apply updates for Microsoft software such as Microsoft SQL Server, SharePoint Server, or .NET framework running on your Windows VMs when you run a patch job. By default, patching these applications is disabled to avoid disruption of service and to separate planned updates for these software. To enable patching Microsoft software automatically, you can either use the Windows UI or the PowerShell.

 $service_manager = New-Object -ComObject 'Microsoft.Update.ServiceManager'
 $service_manager.AddService2("7971f918-a847-4430-9279-4a52d1efe18d",7,"")

Debug a patch job

If your patch fails, you can use the following steps to help find and resolve the issues.

1. Review the instance details for the affected patch job. This helps you to identify which instances failed or what state they are stuck in. The instance details list also contains a brief error message for each instance.

   If a patch fails with a status of NO_AGENT_DETECTED or TIMED_OUT, this usually means that the service sent a request to the agent to begin patching but never heard back from the agent. Review the following possible causes and fixes:

   • The instance is not running. To fix this, start the VM instance.
   • Verify the setup by using the verification checklist.
   • The network settings on the VPC network or the instance didn't allow the OS Config agent to communicate with the OS Config API. To fix this, check the network settings.

2. If the instance details don't provide enough information, review the Cloud Logging logs or serial port console. The OS Config agent writes its log entries to both locations. In Cloud Logging, you can filter using the patch job ID to see all log entries related to that patch job. You can also enable debug logging by setting the osconfig-log-level=debug metadata value at the VM or Google Cloud project level.

What's next?

• Learn more about Patch.
• Manage your patch jobs.
• Schedule patch jobs.