Create and manage vector indexes

This page explains how to create and manage Spanner vector indexes, which use approximate nearest neighbor (ANN) search and tree-based structures to accelerate vector similarity searches on your data.

Spanner accelerates approximate nearest neighbor (ANN) vector searches by using a specialized vector index. This index leverages Google Research's Scalable Nearest Neighbor (ScaNN), a highly efficient nearest neighbor algorithm.

The vector index uses a tree-based structure to partition data and facilitate faster searches. Spanner offers both two-level and three-level tree configurations:

  • Two-level tree configuration: Leaf nodes (num_leaves) contain groups of closely related vectors along with their corresponding centroid. The root level consists of the centroids from all leaf nodes.
  • Three-level tree configuration: Similar in concept to a two-level tree, while introducing an additional branch layer (num_branches), from which leaf node centroids are further partitioned to form the root level (num_leaves).

Spanner picks an index for you. However, if you know that a specific index works best, then you can use the FORCE_INDEX hint to choose to use the most appropriate vector index for your use case.

For more information, see VECTOR INDEX statements.

Limitations

Create vector index

To optimize the recall and performance of a vector index, we recommend that you:

  • Create your vector index after most of the rows with embeddings are written to your database. You might also need to periodically rebuild the vector index after you insert new data. For more information, see Rebuild the vector index.

  • Use the STORING clause to store a copy of a column in the vector index. If a column value is stored in the vector index, then Spanner performs filtering at the index's leaf level to improve query performance. We recommend that you store a column if it's used in a filtering condition. For more information about using STORING in an index, see Create an index for index-only scans.

When you create your table, the embedding column must be an array of the FLOAT32 (recommended) or FLOAT64 data type, and have a vector_length annotation, indicating the dimension of the vectors.

The following DDL statement creates a Documents table with an embedding column DocEmbedding with a vector length:

CREATE TABLE Documents (
  UserId INT64 NOT NULL,
  DocId INT64 NOT NULL,
  Author STRING (1024),
  DocContents Bytes(MAX),
  DocEmbedding ARRAY<FLOAT32>(vector_length=>128) NOT NULL,
  NullableDocEmbedding ARRAY<FLOAT32>(vector_length=>128),
  WordCount INT64,
) PRIMARY KEY (DocId);

After you populate your Documents table, you can create a vector index with a two-level tree and 1000 leaf nodes on the Documents table with an embedding column DocEmbedding using the cosine distance:

CREATE VECTOR INDEX DocEmbeddingIndex
  ON Documents(DocEmbedding)
  STORING (WordCount)
  OPTIONS (distance_type = 'COSINE', tree_depth = 2, num_leaves = 1000);

If your embedding column isn't marked as NOT NULL in the table definition, you must declare it with a WHERE COLUMN_NAME IS NOT NULL clause in the vector index definition, where COLUMN_NAME is the name of your embedding column. To create a vector index with a three-level tree and 1000000 leaf nodes on the nullable embedding column NullableDocEmbedding using the cosine distance:

CREATE VECTOR INDEX DocEmbeddingThreeLevelIndex
  ON Documents(NullableDocEmbedding)
  STORING (WordCount)
  WHERE NullableDocEmbedding IS NOT NULL
  OPTIONS (distance_type = 'COSINE', tree_depth = 3, num_branches=1000, num_leaves = 1000000);

Filter a vector index

You can also create a filtered vector index to find the most similar items in your database that match the filter condition. A filtered vector index selectively indexes rows that satisfy the specified filter conditions, improving search performance.

In the following example, the table Documents2 has a column called Category. In our vector search, we want to index the "Tech" category so we create a generated column that evaluates to NULL if the category condition isn't met.

CREATE TABLE Documents2 (
  DocId INT64 NOT NULL,
  Category STRING(MAX),
  NullIfFiltered BOOL AS (IF(Category = 'Tech', TRUE, NULL)) HIDDEN,
  DocEmbedding ARRAY<FLOAT32>(vector_length=>128),
) PRIMARY KEY (DocId);

Then, we create a vector index with a filter. The TechDocEmbeddingIndex vector index only indexes documents in the "Tech" category.

CREATE VECTOR INDEX TechDocEmbeddingIndex
  ON Documents2(DocEmbedding)
  STORING(NullIfFiltered)
  WHERE DocEmbedding IS NOT NULL AND NullIfFiltered IS NOT NULL
  OPTIONS (...);

When Spanner runs the following query, which has filters that match the TechDocEmbeddingIndex, it automatically picks and is accelerated by TechDocEmbeddingIndex. The query only searches documents in the "Tech" category. You can also use {@FORCE_INDEX=TechDocEmbeddingIndex} to force Spanner to use TechDocEmbeddingIndex explicitly.

SELECT *
FROM Documents2
WHERE DocEmbedding IS NOT NULL AND NullIfFiltered IS NOT NULL
ORDER BY APPROX_(....)
LIMIT 10;

What's next