Troubleshoot Dataform

This document shows you how to resolve issues with Dataform.

Access to BigQuery is denied

The following error occurs when you trigger a workflow invocation before granting Dataform access to BigQuery:

Access Denied: Project PROJECT_ID: User does not have bigquery.jobs.create permission in project PROJECT_ID.

To resolve this error, grant Dataform access to BigQuery.

Access token for a remote repository is rejected

The following error occurs when your authentication token for a connected third-party repository does not have access to that repository:

The access token for remote repository REPOSITORY_NAME was rejected

To resolve this error, check required permissions in your Git provider and update the Secret Manager authentication token accordingly. For more information about authenticating third-party Git repositories in Dataform, see Connect to a third-party Git repository.

BigQuery quota is exceeded

The following error occurs when the number of API requests that Dataform sends to BigQuery exceeds the BigQuery quota:

Quota exceeded: Your user_method exceeded quota for concurrent api requests
per user per method.

To resolve this error, reduce the number of parallel queries to less than 250 in the following ways:

In Dataform, categorize actions with tags, and execute only selected tags at a time.
In Dataform, introduce dependencies between actions.
In Dataform, divide executions of actions between different Google Cloud projects.

For instructions to resolve this error in BigQuery, see Troubleshoot quota and limit errors.

BigQuery query concurrency limit is exceeded

The following error occurs when the number of concurrent queries executed to BigQuery exceeds the BigQuery query concurrency limit:

Exceeded rate limits: too many concurrent queries for this project_and_region

To resolve this error, reduce the number of parallel queries to less than 250 in the following ways:

In Dataform, categorize actions with tags, and execute only selected tags at a time.
In Dataform, introduce dependencies between actions.
In Dataform, divide executions of actions between different Google Cloud projects.

For instructions to resolve this error in BigQuery, see Troubleshoot quota and limit errors.

BigQuery workflow invocation errors

The following errors occur during execution of a SQL workflow to BigQuery:

Workflow invocation errors that begin with BigQuery error messages.

To resolve these errors, see BigQuery error messages.

Compilation is failing

The following errors occur during compilation due to size or number of compiled queries:

Compilation timed out. Reduce the complexity of your project to ensure it can compile within limits.
Compilation exceeded its allowed heap memory limits. Reduce the complexity of your project to ensure it can compile within limits.
Compilation exceeded its allowed ArrayBuffer or string memory limits. Reduce the complexity of your project to ensure it can compile within limits.

To resolve these errors, follow these steps:

Update Dataform core to the latest version.
Inspect your SQL workflow to identify and reduce inefficiencies.
Reduce the size of SQL queries.

Reduce the amount of JavaScript operations in memory, for example:

config { config {type: "table" }}
js {
    const tooBig = new Uint8Array(110_000_000);
}
SELECT ...

Split the repository.

For more information about Dataform compilation resources limits, see Quotas and limits.

`@dataform/core` dependency errors

The following errors occur during compilation if the dataform-core dependency in package.json is outdated:

Failed to resolve @dataform/core

@dataform/core version should be X.X.X or newer

The @dataform/core dependency is required in the package.json. When you initialize the first workspace in your repository, Dataform automatically populates package.json with the current version of @dataform/core. You need to update @dataform/core to the latest version as it comes out.

To resolve these errors, update @dataform/core to the latest version.

Failure to resolve `dataform.json`

The following error occurs when you initialize a Dataform workspace, but the initialization process fails to install all packages:

Uncaught Error: Failed to resolve dataform.json

To resolve this error, in your workspace, open package.json and click Install packages.

`git+` package targets are not supported

The following error occurs when you define packages in package.json with targets prefixed with git+:

'git+' prefixed package targets are not currently supported. However,
in most cases they can be used via a '.tar.gz' suffixed target instead.

Dataform does not support package targets prefixed with git+.

To resolve this error, generate a tar.gz URL of the package and update the package target in package.json. For more information about installing packages in Dataform, see Install a package in Dataform.

Package installation times out

The following error occurs when the size of packages defined in package.json exceeds the maximum size of NPM dependencies:

API request error: Package installation timed out

To resolve this error, remove redundant packages from package.json. Make sure the package.json file does not contain @dataform/cli, and that the total size of defined NPM dependencies does not exceed 200Mb.

If your release configurations reference Git commitishes, make sure that the package.json files at their targets are valid.

Private package registry cannot be reached

The following error occurs when Dataform authentication for a private package expires:

Permission denied when fetching one or more npm packages. Please verify that
private registry authentication details are valid for each npm registry

To resolve this error, verify that private registry authentication details are valid for each NPM registry. For more information, see Authenticate a private package in Dataform.

Remote repository cannot be reached

The following error occurs due to Git flakiness or when a branch name in the connected third-party Git repository contains a non-ascii or non-English character:

Remote repository REPOSITORY_NAME could not be reached.

To resolve this error, inspect branch names in the connected third-party repository. If a branch name contains a non-ascii or non-English character, delete the branch.

Remote repository cannot be reached: `generic::invalid_argument`

The following error occurs on the Release configurations details page when a scheduled release occasionally encounters a slow, flaky, or dropped GitHub, GitLab, or Bitbucket connection:

generic::invalid_argument: Remote repository 'REMOTE_REPOSITORY_URL' could not be reached.

There is no need to take any action. Unless GitHub, GitLab, or Bitbucket Cloud issues persist, the subsequent scheduled releases can be successful.

Secret for a remote repository cannot be reached

The following error occurs when your Dataform service account cannot access your Secret Manager secret for a connected third-party repository:

Dataform's service account is unable to reach the configured secret.
Make sure the secret exists and is shared with your Dataform service account:
SERVICE_ACCOUNT_ID.

To resolve this error, do the following:

Verify that your Dataform service account has access to the secret.
Exclude the secret from your VPC-SC service perimeter. Dataform does not support VPC-SC at this time.