Node pools are updated in parallel by default. This can potentially
causing two issues:
Quota issue
For each node pool, it will create one surge node, so that N node pools will
create N surge nodes when they are updated in parallel. This may result in
resource exhaustion if there is limited extra capacity for creating those
surge nodes.
PDB deadlock issue
Draining more than one node at a time could potentially cause PDB deadlocks.
This doc shows how to specify a node pool update policy to configure the
maximum number of nodes to be updated concurrently during node pool update,
which can help avoid the two issues mentioned. In Anthos 1.13, we restrict
the number to be either 0 or 1. 0 means preserving the current parallel behavior
, which is the default value. 1 means updating the node pools sequentially.
Note that while sequential update can avoid the two issues, it could make the
node pool update process take longer time than updating in parallel.
Configure node pool update policy
In user cluster seed config user-cluster.yaml, you can configure the node pool
update to be done sequentially as follows:
Note that this value for maximumConcurrentNodePoolUpdate is only allowed to
be either 0 or 1. The sequential behavior applies to both node pool creation and
update. Also in case any node pool
update/create encountered issues and got stuck, the current behavior is that we
will block there and won't move on to update next node pool.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-04-02 UTC."],[[["Pre-GA features, like the node pool update policy, are subject to the Pre-GA Offerings Terms and are available \"as is\" with potentially limited support."],["Updating node pools in parallel can cause quota issues due to the creation of surge nodes and potential PDB deadlocks from draining multiple nodes simultaneously."],["The node pool update policy allows configuration of the maximum number of nodes to be updated concurrently, with options limited to either 0 for parallel updates or 1 for sequential updates in Anthos 1.13."],["Sequential updates, while avoiding quota and PDB deadlock issues, may increase the overall duration of the node pool update process compared to parallel updates."],["Configuring the sequential node pool update is done using the `maximumConcurrentNodePoolUpdate` setting in user-cluster.yaml, with the value of 1, which blocks further node pool updates/creations if an issue occurs."]]],[]]
