REST Resource: projects.locations.dataScans

Resource: DataScan

Represents a user-visible job which provides the insights for the related data source.

For example:

  • Data quality: generates queries based on the rules and runs against the data to get data quality check results. For more information, see Auto data quality overview.
  • Data profile: analyzes the data in tables and generates insights about the structure, content and relationships (such as null percent, cardinality, min/max/mean, etc). For more information, see About data profiling.
  • Data discovery: scans data in Cloud Storage buckets to extract and then catalog metadata. For more information, see Discover and catalog Cloud Storage data.
  • Data documentation: analyzes the table details and generates insights including descriptions and sample SQL queries for the table. For more information, see Generate data insights in BigQuery.
JSON representation 
{
  "name": string,
  "uid": string,
  "description": string,
  "displayName": string,
  "labels": {
    string: string,
    ...
  },
  "state": enum (State),
  "createTime": string,
  "updateTime": string,
  "data": {
    object (DataSource)
  },
  "executionSpec": {
    object (ExecutionSpec)
  },
  "executionStatus": {
    object (ExecutionStatus)
  },
  "type": enum (DataScanType),

  // Union field spec can be only one of the following:
  "dataQualitySpec": {
    object (DataQualitySpec)
  },
  "dataProfileSpec": {
    object (DataProfileSpec)
  },
  "dataDiscoverySpec": {
    object (DataDiscoverySpec)
  },
  "dataDocumentationSpec": {
    object (DataDocumentationSpec)
  }
  // End of list of possible types for union field spec.

  // Union field result can be only one of the following:
  "dataQualityResult": {
    object (DataQualityResult)
  },
  "dataProfileResult": {
    object (DataProfileResult)
  },
  "dataDiscoveryResult": {
    object (DataDiscoveryResult)
  },
  "dataDocumentationResult": {
    object (DataDocumentationResult)
  }
  // End of list of possible types for union field result.
}
Fields
name

string

Output only. Identifier. The relative resource name of the scan, of the form: projects/{project}/locations/{locationId}/dataScans/{datascanId}, where project refers to a projectId or project_number and locationId refers to a Google Cloud region.
uid

string

Output only. System generated globally unique ID for the scan. This ID will be different if the scan is deleted and re-created with the same name.
description

string

Optional. Description of the scan.

  • Must be between 1-1024 characters.
displayName

string

Optional. User friendly display name.

  • Must be between 1-256 characters.
labels

map (key: string, value: string)

Optional. User-defined labels for the scan.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
state

enum (State)

Output only. Current state of the DataScan.
createTime

string (Timestamp format)

Output only. The time when the scan was created.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
updateTime

string (Timestamp format)

Output only. The time when the scan was last updated.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
data

object (DataSource)

Required. The data source for DataScan.
executionSpec

object (ExecutionSpec)

Optional. DataScan execution settings.

If not specified, the fields in it will use their default values.
executionStatus

object (ExecutionStatus)

Output only. Status of the data scan execution.
type

enum (DataScanType)

Output only. The type of DataScan.
Union field spec. Data scan related setting. The settings are required and immutable. After you configure the settings for one type of data scan, you can't change the data scan to a different type of data scan. spec can be only one of the following:
dataQualitySpec

object (DataQualitySpec)

Settings for a data quality scan.
dataProfileSpec

object (DataProfileSpec)

Settings for a data profile scan.
dataDiscoverySpec

object (DataDiscoverySpec)

Settings for a data discovery scan.
dataDocumentationSpec

object (DataDocumentationSpec)

Settings for a data documentation scan.
Union field result. The result of the data scan. result can be only one of the following:
dataQualityResult

object (DataQualityResult)

Output only. The result of a data quality scan.
dataProfileResult

object (DataProfileResult)

Output only. The result of a data profile scan.
dataDiscoveryResult

object (DataDiscoveryResult)

Output only. The result of a data discovery scan.
dataDocumentationResult

object (DataDocumentationResult)

Output only. The result of a data documentation scan.

DataSource

The data source for DataScan.

JSON representation 
{

  // Union field source can be only one of the following:
  "entity": string,
  "resource": string
  // End of list of possible types for union field source.
}
Fields
Union field source. The source is required and immutable. Once it is set, it cannot be change to others. source can be only one of the following:
entity

string

Immutable. The Dataplex Universal Catalog entity that represents the data source (e.g. BigQuery table) for DataScan, of the form: projects/{project_number}/locations/{locationId}/lakes/{lakeId}/zones/{zoneId}/entities/{entityId}.
resource

string

Immutable. The service-qualified full resource name of the cloud resource for a DataScan job to scan against. The field could either be: Cloud Storage bucket for DataDiscoveryScan Format: //storage.googleapis.com/projects/PROJECT_ID/buckets/BUCKET_ID or BigQuery table of type "TABLE" for DataProfileScan/DataQualityScan/DataDocumentationScan Format: //bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID

ExecutionSpec

DataScan execution settings.

JSON representation 
{
  "trigger": {
    object (Trigger)
  },

  // Union field incremental can be only one of the following:
  "field": string
  // End of list of possible types for union field incremental.
}
Fields
trigger

object (Trigger)

Optional. Spec related to how often and when a scan should be triggered.

If not specified, the default is OnDemand, which means the scan will not run until the user calls dataScans.run API.

Union field incremental. Spec related to incremental scan of the data

When an option is selected for incremental scan, it cannot be unset or changed. If not specified, a data scan will run for all data in the table. incremental can be only one of the following:
field

string

Immutable. The unnested field (of type Date or Timestamp) that contains values which monotonically increase over time.

If not specified, a data scan will run for all data in the table.

Trigger

DataScan scheduling and trigger settings.

JSON representation 
{

  // Union field mode can be only one of the following:
  "onDemand": {
    object (OnDemand)
  },
  "schedule": {
    object (Schedule)
  }
  // End of list of possible types for union field mode.
}
Fields

Union field mode. DataScan scheduling and trigger settings.

If not specified, the default is onDemand. mode can be only one of the following:
onDemand

object (OnDemand)

The scan runs once via dataScans.run API.
schedule

object (Schedule)

The scan is scheduled to run periodically.

OnDemand

This type has no fields.

The scan runs once via dataScans.run API.

Schedule

The scan is scheduled to run periodically.

JSON representation 
{
  "cron": string
}
Fields
cron

string

Required. Cron schedule for running scans periodically.

To explicitly set a timezone in the cron tab, apply a prefix in the cron tab: "CRON_TZ=${IANA_TIME_ZONE}" or "TZ=${IANA_TIME_ZONE}". The ${IANA_TIME_ZONE} may only be a valid string from IANA time zone database (wikipedia). For example, CRON_TZ=America/New_York 1 * * * *, or TZ=America/New_York 1 * * * *.

This field is required for Schedule scans.

ExecutionStatus

Status of the data scan execution.

JSON representation 
{
  "latestJobStartTime": string,
  "latestJobEndTime": string,
  "latestJobCreateTime": string
}
Fields
latestJobStartTime

string (Timestamp format)

Optional. The time when the latest DataScanJob started.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
latestJobEndTime

string (Timestamp format)

Optional. The time when the latest DataScanJob ended.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
latestJobCreateTime

string (Timestamp format)

Optional. The time when the DataScanJob execution was created.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".

Methods

create

 Creates a DataScan resource.

delete

 Deletes a DataScan resource.

generateDataQualityRules

 Generates recommended data quality rules based on the results of a data profiling scan.

get

 Gets a DataScan resource.

getIamPolicy

 Gets the access control policy for a resource.

list

 Lists DataScans.

patch

 Updates a DataScan resource.

run

 Runs an on-demand execution of a DataScan

setIamPolicy

 Sets the access control policy on the specified resource.

testIamPermissions

 Returns permissions that a caller has on the specified resource.