VertexAIIndex

Property	Value
Google Cloud Service Name	Vertex AI
Google Cloud Service Documentation	/vertex-ai/docs
Google Cloud REST Resource Name	v1beta1.projects.locations.indexes
Google Cloud REST Resource Documentation	/vertex-ai/docs/reference/rest/v1beta1/projects.locations.indexes
Config Connector Resource Short Names	gcpvertexaiindex gcpvertexaiindexes vertexaiindex
Config Connector Service Name	aiplatform.googleapis.com
Config Connector Resource Fully Qualified Name	vertexaiindexes.vertexai.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember	No
Config Connector Default Average Reconcile Interval In Seconds	600

Custom Resource Definition Properties

Spec

Schema

description: string
displayName: string
indexUpdateMethod: string
metadata:
  config:
    algorithmConfig:
      bruteForceConfig: {}
      treeAhConfig:
        leafNodeEmbeddingCount: integer
        leafNodesToSearchPercent: integer
    approximateNeighborsCount: integer
    dimensions: integer
    distanceMeasureType: string
    featureNormType: string
    shardSize: string
  contentsDeltaUri: string
projectRef:
  external: string
  name: string
  namespace: string
region: string
resourceID: string

Fields

Fields
`description` Optional	`string` The description of the Index.
`displayName` Required	`string` The display name of the Index. The name can be up to 128 characters long and can consist of any UTF-8 characters.
`indexUpdateMethod` Optional	`string` Immutable. The update method to use with this Index. The value must be the followings. If not set, BATCH_UPDATE will be used by default. * BATCH_UPDATE: user can call indexes.patch with files on Cloud Storage of datapoints to update. * STREAM_UPDATE: user can call indexes.upsertDatapoints/DeleteDatapoints to update the Index and the updates will be applied in corresponding DeployedIndexes in nearly real-time.
`metadata` Optional	`object` An additional information about the Index.
`metadata.config` Optional	`object` Immutable. The configuration of the Matching Engine Index.
`metadata.config.algorithmConfig` Optional	`object` The configuration with regard to the algorithms used for efficient search.
`metadata.config.algorithmConfig.bruteForceConfig` Optional	`object` Configuration options for using brute force search, which simply implements the standard linear search in the database for each query.
`metadata.config.algorithmConfig.treeAhConfig` Optional	`object` Configuration options for using the tree-AH algorithm (Shallow tree + Asymmetric Hashing). Please refer to this paper for more details: https://arxiv.org/abs/1908.10396.
`metadata.config.algorithmConfig.treeAhConfig.leafNodeEmbeddingCount` Optional	`integer` Number of embeddings on each leaf node. The default value is 1000 if not set.
`metadata.config.algorithmConfig.treeAhConfig.leafNodesToSearchPercent` Optional	`integer` The default percentage of leaf nodes that any query may be searched. Must be in range 1-100, inclusive. The default value is 10 (means 10%) if not set.
`metadata.config.approximateNeighborsCount` Optional	`integer` The default number of neighbors to find via approximate search before exact reordering is performed. Exact reordering is a procedure where results returned by an approximate search algorithm are reordered via a more expensive distance computation. Required if tree-AH algorithm is used.
`metadata.config.dimensions` Required*	`integer` The number of dimensions of the input vectors.
`metadata.config.distanceMeasureType` Optional	`string` The distance measure used in nearest neighbor search. The value must be one of the followings: * SQUARED_L2_DISTANCE: Euclidean (L_2) Distance * L1_DISTANCE: Manhattan (L_1) Distance * COSINE_DISTANCE: Cosine Distance. Defined as 1 - cosine similarity. * DOT_PRODUCT_DISTANCE: Dot Product Distance. Defined as a negative of the dot product.
`metadata.config.featureNormType` Optional	`string` Type of normalization to be carried out on each vector. The value must be one of the followings: * UNIT_L2_NORM: Unit L2 normalization type * NONE: No normalization type is specified.
`metadata.config.shardSize` Optional	`string` Immutable. Index data is split into equal parts to be processed. These are called "shards". The shard size must be specified when creating an index. The value must be one of the followings: * SHARD_SIZE_SMALL: Small (2GB) * SHARD_SIZE_MEDIUM: Medium (20GB) * SHARD_SIZE_LARGE: Large (50GB).
`metadata.contentsDeltaUri` Optional	`string` Allows creating or replacing the contents of the Matching Engine Index. When being updated, the existing content of the Index will be replaced by the data from the latest contentsDeltaUri. The string must be a valid Cloud Storage directory path. If this field is set when calling IndexService.UpdateIndex, then no other Index field can be also updated as part of the same call. The expected structure and format of the files this URI points to is described at https://cloud.google.com/vertex-ai/docs/matching-engine/using-matching-engine#input-data-format.
`projectRef` Required	`object` The project that this resource belongs to.
`projectRef.external` Optional	`string` Allowed value: The `name` field of a `Project` resource.
`projectRef.name` Optional	`string` Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
`projectRef.namespace` Optional	`string` Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
`region` Required	`string` Immutable. The region of the index. eg us-central1.
`resourceID` Optional	`string` Immutable. Optional. The service-generated name of the resource. Used for acquisition only. Leave unset to create a new resource.

description

Optional

string

The description of the Index.

displayName

Required

string

The display name of the Index. The name can be up to 128 characters long and can consist of any UTF-8 characters.

indexUpdateMethod

Optional

string

Immutable. The update method to use with this Index. The value must be the followings. If not set, BATCH_UPDATE will be used by default. * BATCH_UPDATE: user can call indexes.patch with files on Cloud Storage of datapoints to update. * STREAM_UPDATE: user can call indexes.upsertDatapoints/DeleteDatapoints to update the Index and the updates will be applied in corresponding DeployedIndexes in nearly real-time.

metadata

Optional

object

An additional information about the Index.

metadata.config

Optional

object

Immutable. The configuration of the Matching Engine Index.

metadata.config.algorithmConfig

Optional

object

The configuration with regard to the algorithms used for efficient search.

metadata.config.algorithmConfig.bruteForceConfig

Optional

object

Configuration options for using brute force search, which simply implements the standard linear search in the database for each query.

metadata.config.algorithmConfig.treeAhConfig

Optional

object

Configuration options for using the tree-AH algorithm (Shallow tree + Asymmetric Hashing). Please refer to this paper for more details: https://arxiv.org/abs/1908.10396.

metadata.config.algorithmConfig.treeAhConfig.leafNodeEmbeddingCount

Optional

integer

Number of embeddings on each leaf node. The default value is 1000 if not set.

metadata.config.algorithmConfig.treeAhConfig.leafNodesToSearchPercent

Optional

integer

The default percentage of leaf nodes that any query may be searched. Must be in range 1-100, inclusive. The default value is 10 (means 10%) if not set.

metadata.config.approximateNeighborsCount

Optional

integer

The default number of neighbors to find via approximate search before exact reordering is performed. Exact reordering is a procedure where results returned by an approximate search algorithm are reordered via a more expensive distance computation. Required if tree-AH algorithm is used.

metadata.config.dimensions

Required*

integer

The number of dimensions of the input vectors.

metadata.config.distanceMeasureType

Optional

string

The distance measure used in nearest neighbor search. The value must be one of the followings: * SQUARED_L2_DISTANCE: Euclidean (L_2) Distance * L1_DISTANCE: Manhattan (L_1) Distance * COSINE_DISTANCE: Cosine Distance. Defined as 1 - cosine similarity. * DOT_PRODUCT_DISTANCE: Dot Product Distance. Defined as a negative of the dot product.

metadata.config.featureNormType

Optional

string

Type of normalization to be carried out on each vector. The value must be one of the followings: * UNIT_L2_NORM: Unit L2 normalization type * NONE: No normalization type is specified.

metadata.config.shardSize

Optional

string

Immutable. Index data is split into equal parts to be processed. These are called "shards". The shard size must be specified when creating an index. The value must be one of the followings: * SHARD_SIZE_SMALL: Small (2GB) * SHARD_SIZE_MEDIUM: Medium (20GB) * SHARD_SIZE_LARGE: Large (50GB).

metadata.contentsDeltaUri

Optional

string

Allows creating or replacing the contents of the Matching Engine Index. When being updated, the existing content of the Index will be replaced by the data from the latest contentsDeltaUri. The string must be a valid Cloud Storage directory path. If this field is set when calling IndexService.UpdateIndex, then no other Index field can be also updated as part of the same call. The expected structure and format of the files this URI points to is described at https://cloud.google.com/vertex-ai/docs/matching-engine/using-matching-engine#input-data-format.

projectRef

Required

object

The project that this resource belongs to.

projectRef.external

Optional

string

Allowed value: The `name` field of a `Project` resource.

projectRef.name

Optional

string

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

projectRef.namespace

Optional

string

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

region

Required

string

Immutable. The region of the index. eg us-central1.

resourceID

Optional

string

Immutable. Optional. The service-generated name of the resource. Used for acquisition only. Leave unset to create a new resource.

* Field is required when parent field is specified

Status

Schema

conditions:
- lastTransitionTime: string
  message: string
  reason: string
  status: string
  type: string
observedGeneration: integer
observedState:
  createTime: string
  indexStats:
  - shardsCount: integer
    vectorsCount: string
  metadataSchemaUri: string
  name: string

Fields
`conditions`	`list (object)` Conditions represent the latest available observation of the resource's current state.
`conditions[]`	`object`
`conditions[].lastTransitionTime`	`string` Last time the condition transitioned from one status to another.
`conditions[].message`	`string` Human-readable message indicating details about last transition.
`conditions[].reason`	`string` Unique, one-word, CamelCase reason for the condition's last transition.
`conditions[].status`	`string` Status is the status of the condition. Can be True, False, Unknown.
`conditions[].type`	`string` Type is the type of the condition.
`observedGeneration`	`integer` ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.
`observedState`	`object` The observed state of the underlying GCP resource.
`observedState.createTime`	`string` The timestamp of when the Index was created in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits.
`observedState.indexStats`	`list (object)` Stats of the index resource.
`observedState.indexStats[]`	`object`
`observedState.indexStats[].shardsCount`	`integer` The number of shards in the Index.
`observedState.indexStats[].vectorsCount`	`string` The number of vectors in the Index.
`observedState.metadataSchemaUri`	`string` Points to a YAML file stored on Google Cloud Storage describing additional information about the Index, that is specific to it. Unset if the Index does not have any additional information.
`observedState.name`	`string` The resource name of the Index.

Sample YAML(s)

Typical Use Case

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: vertexai.cnrm.cloud.google.com/v1beta1
kind: VertexAIIndex
metadata:
  labels:
    label-one: "value-one"
  name: vertexaiindex-sample
spec:
  displayName: "vertex AI index"
  region: us-central1
  description: "an example vertex AI index"
  metadata:
    # Replace ${KCC_VERTEX_AI_INDEX_TEST_DATA_URI?} with the Cloud Storage
    # directory path to the files for inserting, updating or deleting the
    # contents of the Matching Engine Index. For example,
    # "gs://my-test-bucket/contents"
    contentsDeltaUri: ${KCC_VERTEX_AI_INDEX_TEST_DATA_URI?}
    config:
      dimensions: 2
      approximateNeighborsCount: 150
      shardSize: "SHARD_SIZE_SMALL"
      distanceMeasureType: "DOT_PRODUCT_DISTANCE"
      algorithmConfig:
        treeAhConfig:
          leafNodeEmbeddingCount: 500
          leafNodesToSearchPercent: 7
  indexUpdateMethod: "BATCH_UPDATE"
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID.
    external: ${PROJECT_ID?}
---
# Replace ${PROJECT_NUMBER?} below with your desired project number.
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: vertexaiindex-dep
spec:
  memberFrom:
    serviceIdentityRef:
      name: vertexaiindex-dep
  role: roles/storage.admin # required by vertex AI service agent to access test data
  resourceRef:
    apiVersion: storage.cnrm.cloud.google.com/v1beta1
    kind: StorageBucket
    external: ${KCC_VERTEX_AI_INDEX_TEST_BUCKET?}
---
apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
kind: ServiceIdentity
metadata:
  name: vertexaiindex-dep
  annotations:
    cnrm.cloud.google.com/deletion-policy: "abandon"
spec:
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID.
    external: ${PROJECT_ID?}
  resourceID: aiplatform.googleapis.com