This page describes Config Sync error codes and recommended actions to handle these errors.
Config Sync error messages consist of an error ID in the format KNV1234
where 1234 is a unique number, followed by a description of the problem and a
suggestion for how to fix it. K is inherited from Kubernetes conventions,
rules with prefix N are specific to nomos, V is specific to
errors detectable in the initial state of the repository and the cluster. Codes
for errors detectable in the initial state of the repository and the cluster are
of the form KNV1XXX. Codes for errors that can only be detected at runtime are
of the form KNV2XXX.
KNV error table
| Error code | Description | Recommended action |
|---|---|---|
|
The ID of InternalError changed to |
N/A |
|
Deprecated in Config Sync 1.3. |
N/A |
|
Deprecated in Config Sync 1.3. |
N/A |
|
When using a hierarchical repository structure, a directory that contains a namespace config must not contain any subdirectories. A directory without a namespace config is an abstract namespace directory and has directories inheriting from it. Consequently, abstract namespace directories must have subdirectories. A directory containing a namespace config is a namespace directory and cannot be inherited from, so it must not have any subdirectories. |
Either remove the namespace config from the parent directory, or move the subdirectory somewhere else. |
|
A cluster-scoped object must not declare the annotation
|
Remove |
|
The only valid setting for the management annotation is
|
Ensure that the management annotation is
For more information, see Managing objects. |
|
An object declared in the repository couldn't be parsed. |
Validate your YAML format. For example, you can use the
If |
|
When using an unstructured repository, configs must not be declared in an abstract namespace directory. |
Move the config listed in the error message to a namespace directory. For more information, see Using an unstructured repository. |
|
When using a hierarchical repository structure, configs must either declare namespaces that match the namespace directory containing them or omit the field. |
Update the namespace field identified in the error message. For more information, see Structure of the hierarchical repository. |
|
Configs must not declare unsupported annotations starting with |
Make sure you're using one of the following supported annotations:
|
|
Configs must not have labels with keys that begin with
|
Update any labels identified in the error message. For example, if
you tried to declare a label named |
|
Deprecated in Config Sync 1.3. |
N/A |
|
The config refers to a ClusterSelector or NamespaceSelector that doesn't exist. Before you can use a selector in an annotation for a config, the selector must exist. |
Either create any missing selectors, or, if the selector was removed, remove any configs that refer to it. |
|
ClusterSelector and NamespaceSelector configs use the correct syntax, but a syntax error was found. |
Ensure that you specify the config using the appropriate data schema: |
|
Deprecated in Config Sync 1.3.2. | N/A |
|
When using the hierarchical repository structure, a config for the ConfigManagement Operator
must exist in the |
Define at least a minimal config for the ConfigManagement Operator. For more information, see Structure of the hierarchical repository. |
|
Deprecated in Config Sync 1.3. | N/A |
|
When using the hierarchical repository structure, namespaces mustn't be declared directly
in the |
Create a subdirectory for the namespace configs listed in the error message. For more information, see Structure of the hierarchical repository. |
|
When using a hierarchical repository structure, a namespace config declares
metadata.name, and its value must match the name of the namespace's directory.
|
Correct the namespace's metadata.name or its directory. |
|
No CustomResourceDefinition is defined for the resource in the cluster. |
Create a CustomResourceDefinition for the resource referenced in the error message. Resource types that are not built-in Kubernetes objects must have a CustomResourceDefinition. |
|
When you're using a hierarchical repository, configs of this Kind cannot be declared in
the |
Move the resource referenced in the error message out of the |
|
The |
If your repository's format is compatible with the supported version, update the
|
|
Directory names must have fewer than 64 characters, consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character. |
Rename or remove the misnamed directory. |
|
Configs of the same Kind must have unique names in the same namespace and their parent abstract namespaces. |
Rename or remove any configs referenced in the error message so that they all have unique names. |
|
Multiple namespace resources cannot exist in the same directory. |
Remove the duplicate config(s) so that no more than one namespace resource remains. |
|
All configs must declare |
Add the |
|
The type |
Remove the problematic config, or convert your repository to use
|
|
If you're using a hierarchical repository, you can only declare
|
Ensure any configs declared in the |
|
It is forbidden to declare the |
If you declared the
If you declared the
If you were previously syncing to a hierarchical repository and had to declare the controller namespace alongside any resources, consider switching to an unstructured repository for more flexibility in your source structure. |
|
The |
Change the
If |
|
Deprecated in Config Sync 1.3. | N/A |
|
It is forbidden to declare a namespace-scoped object outside of the
|
Move the problematic configs so that they are in a legal directory. For more information about namespace-scoped objects, see Namespace-scoped objects. |
|
It is forbidden to declare a cluster-scoped object outside of the |
Move the problematic configs so that they are in a legal directory. For more information about cluster-scoped objects, see Cluster-scoped objects. |
|
Deprecated in Config Sync 1.3. | N/A |
|
This Resource Kind must not be declared in a |
Remove the problematic resource. To read more about |
|
An illegal value for |
Change |
|
Config Sync cannot configure this object. |
Remove the problematic config from the repository. |
|
An abstract namespace directory with configs must have at least one namespace subdirectory. |
Either add a namespace directory under your abstract namespace directory, add a namespace config to your abstract namespace directory, or remove the configs in your abstract namespace directory. |
|
Configs with |
Remove the |
|
This |
Update the |
|
Removing a custom resource definition (CRD) and leaving the corresponding Custom Resources in the repository isn't allowed. |
Remove the CRD along with the Custom Resources. |
|
The CustomResourceDefinition has an invalid name. |
Change the name to the recommendation in the error message. |
|
The config is using a deprecated Group and Kind. |
Change the Group or Kind to the recommendation in the error message. |
|
Cluster-scoped resources mustn't declare |
Remove the metadata.namespace field from your cluster-scoped resource. |
|
Namespace-scoped resources must either declare either |
Add the missing field to your namespace-scoped resource. |
|
Configs contain an invalid value for an annotation. |
Follow the instructions in the error message to resolve the error. |
|
The value of |
Update the value of
|
|
A resource is declared in an unmanaged namespace. |
Either remove the |
|
A resource has an illegal label. |
Remove the illegal labels listed in the error message. |
|
A namespace repository can only declare namespace-scoped resources in the namespace the repository it applies to. |
Make sure all namespace repositories are declaring namespace-scoped resources correctly.
For example, the repository for the
For example, if a config in the In addition to making sure that namespace-scoped resources are declared correctly, make sure that namespaces are declared in the root repository. This is necessary because namespaces are cluster-scoped. |
|
A namespace repository can declare at most one Kptfile resource. |
Remove all but one Kptfile resource. |
|
When managing objects in multiple sources of truth, conflicts can arise when the same object (matching group, kind, name, and namespace) is declared in more than one source. For example, when the same object is managed by a RootSync and a RepoSync, the RootSync wins. If the RootSync applies first, the RepoSync reports a KNV1060 status error. If the RepoSync applies first, the RootSync overwrites the RepoSync's object and the RepoSync reports a KNV1060 status error when it sees the update. |
Resolve the conflict by updating the config to match with the other source of truth, or by deleting the conflicting object from one of the sources.
The |
|
An |
Follow the instructions in the error message to fix the configuration errors. |
|
The Kptfile doesn't have a valid inventory field. A Kptfile should have a non-empty inventory field with both identifier and namespace specified. |
Specify the values for |
|
Kptfiles were found in the Root repository. Kptfiles are only supported in namespace-scoped repositories. |
Remove the Kptfiles from the Root repository. |
|
The |
Follow the instructions in the error message. For example, you might need to re-run
|
|
The CustomResourceDefinition is malformed. |
Check the field specified by the error message and make sure its value is correctly formatted. |
|
A config object must declare only cluster-selector annotation. This error occurs when both
the legacy annotation ( |
Remove one of the annotations from the |
|
The reconciler fails to encode the declared fields into a format that is compatible with server-side apply. It could be caused by an out-of-date schema. |
Check the field specified by the error message and make sure it matches the schema of the resource kind. |
|
The rendering process encountered a user-actionable issue. |
If the Git repository contains Kustomize configurations, but no
If the error is caused by
If a |
|
A reconciler has reconciled its own RootSync or RepoSync object. A RootSync object can manage other RootSync and RepoSync objects; A RepoSync object can manage other RepoSync objects, but they cannot self-manage. |
Remove the RootSync or RepoSync object from the source of truth that the object syncs from. |
|
An OS-level system call accessing a file system resource fails. |
This error is likely caused by an invalid YAML configuration or using special characters. If you
have an invalid YAML configuration, you see an error message similar to the following:
If your filename or path contains special characters, you might see an error message
similar to |
|
A request accessing the Kubernetes API Server fails. |
Kubernetes API requests can fail for many reasons. Common causes include:
Config Sync retries after most API Server errors. Some might be temporary issues that resolve themselves, but most will require user intervention to resolve. API Server errors are rarely caused by Config Sync itself, but if you suspect they might be, please file a bug report. |
|
A generic OS-level system call fails. | |
|
Config Sync cannot read from the source of truth. |
Multiple issues can cause this error. To troubleshoot connecting to the
source of truth, see
Troubleshoot connecting to the source of truth.
To learn about known issues that cause |
|
Config Sync is fighting with another controller over a resource. Such fights consume a high amount of resources and can degrade your performance. | For tips on how to diagnose and resolve controller fights, see Troubleshoot controller fights. |
|
To help prevent accidental deletion, Config Sync doesn't let you remove all namespaces or cluster-scoped resources in a single commit. |
If the Config Sync admission webhook is disabled, revert the commit that deletes all resources.
If you want to delete the full set of resources under management, complete the following steps:
|
|
A resource on the API Server is modified or deleted while Config Sync is also attempting to modify it. |
If this type of error only appears at startup or infrequently, you can ignore these errors.
If these errors are not transient (persisting multiple minutes), it might indicate a
serious issue and |
|
This is a generic error indicating that Config Sync failed to sync some configs to the cluster. |
There are multiple issues that can cause this error. For tips on how to resolve common issues with syncing, see Troubleshoot syncing. |
|
This is a generic error indicating a problem with a resource or set of resources. |
The error message includes the specific resources which caused the error. Investigate these resources. |
|
A specific resource is required to proceed, but the resource wasn't found. For example, ConfigManagement Operator tried to update a resource, but the resource was deleted while calculating the update. |
Create or restore the missing resource. |
|
This error reports that more than one instance of an APIResource was found in a context
where exactly one of that APIResource is allowed. For example, only one |
Remove the additional APIResource. |
|
A namespace reconciler has insufficient permissions to manage resources. |
Make sure the reconciler has sufficient permissions. |
|
This warning occurs when the Config Sync webhook configuration is illegally modified. The illegal webhook configurations are ignored. |
Remove the illegally modified webhook. |
|
The rendering process encountered an internal issue. For example, Config Sync is unable to access the file system. |
This error might indicate the Pod is not healthy. You can restart the reconciler Pods by running the following commands: # restart a root reconciler kubectl delete pod -n config-management-system -l configsync.gke.io/reconciler=root-reconciler # restart a namespace reconciler kubectl delete pod -n config-management-system -l configsync.gke.io/reconciler=ns-reconciler-NAMESPACE |
|
This error represents a transient issue that should automatically resolve at a later time. For example, if the rendering state doesn't match the source config, you might see this error. |
The error should automatically resolve. |
|
There's a problem with Config Sync itself. |
Please file a bug report. |
|
You've encountered an error with no documented error message. |
We haven't yet written documentation specific to the error you encountered. |
Error messages without a KNV code
Errors that are reported by Config Sync reconcilers have the KNV error code, but errors reported from other components don't have KNV code. For example, the permission denied error comes from the fleet controller, which is a layer on top of Config Sync.
The following table lists some common errors without the KNV prefix.
| Error message | Recommended action |
|---|---|
|
|
Cannot build exporters
When a component in Open Telemetry Collector can't access the
default service account under the same namespace, you might notice
that the This issue usually happens when Workload Identity Federation for GKE is enabled in a cluster. To solve this problem, follow the instructions in Monitoring Config Sync to grant metric write permission to the default service account.
If the error persists after setting up the IAM, restart the
|
server certificate verification failed. CAfile:/etc/ca-cert/cert CRLfile: none
|
Server certificate verification failed
If the This message indicates that the server is configured with certificates from a custom Certificate Authority (CA). However, the custom CA is not properly configured, resulting in the failure of the container to fetch from the server.
To resolve this issue, first, you can verify whether the
Next, if the field has been configured and the Secret object exists,
ensure that the Secret object contains the full certificates. Here is an example of how to list the server certificates: echo -n | openssl s_client -showcerts -connect HOST:PORT -servername SERVER_NAME 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' You can request that your network administration team obtain the CA certificates for you. |
Error message: "MESSAGE": "Unable to retrieve pull secret, the image pull may not succeed."
|
Unable to retrieve pull secret, the image pull may not succeedIf you're using a private registry with Google Distributed Cloud, the Config Sync installation or upgrade can get stuck. You'll see an error similar to this message. To resolve this issue, follow the steps in Update Config Sync using a private registry before installing or upgrading Config Sync. |
Permission 'gkehub.features.create' denied on 'projects/PROJECT_ID/locations/global/features/configmanagement'
|
Permission deniedIf you receive an error similar to this example when you try to configure Config Sync, you might not have the GKE Hub Admin role. To ensure that you have the required permissions, make sure you have granted the required IAM roles. |
What's next
- View known issues.
- Read the introduction to troubleshooting Config Sync.
- If you need additional support, reach out to Cloud Customer Care.