Measure vector query recall

Stay organized with collections Save and categorize content based on your preferences.

This page describes how to measure vector query recall in AlloyDB for PostgreSQL. Recall is the percentage of nearest neighbors returned by the index that are actually true nearest neighbors. For example, if a nearest neighbor query for the 20 nearest neighbors returned 19 of the ground truth nearest neighbors, the recall is 19/20x100 = 95%.

In a vector query, recall is important because it measures the percentage of relevant results retrieved from a search. Recall helps you evaluate the accuracy of the results from an ANN search as compared to the results from a KNN search, which are slower.

This page assumes that you're familiar with PostgreSQL, AlloyDB, and vector search.

Before you begin

1. To store generated embeddings as vector values, install the vector extension version 0.8.0.google-2 or later, which includes pgvector functions and operators. Google extends this version of pgvector with optimizations specific to AlloyDB.

   CREATE EXTENSION IF NOT EXISTS vector;
   

   For more information, see Store, index, and query vectors.

2. To generate ScaNN indexes, install the alloydb_scann extension and the vector extension.

   CREATE EXTENSION IF NOT EXISTS alloydb_scann;
   

Evaluate recall for vector queries on a ScaNN index

You can find the recall for a vector query on a ScaNN index for a given configuration using the evaluate_query_recall function. This function lets you tune your parameters to achieve the vector query recall results that you want. Recall is the metric used for search quality, and is defined as the percentage of the returned results that are objectively closest to the query vectors.

Find the recall for a vector query

1. Open a SQL editor in AlloyDB Studio or open a psql client.
2. Create the alloydb_scann extension and then create an index using the ScaNN index method.

3. Execute the evaluate_query_recall function, which takes in the query as a parameter and returns the following recall:

   SELECT * FROM evaluate_query_recall( QUERY_TEXT, QUERY_TIME_CONIGURATIONS )
   

   Before you run this command, make the following replacements:

   • QUERY_TEXT: the text of the query.
   • QUERY_TIME_CONFIGURATIONS: the configuration that you can set for the ANN query. This must be in a JSON format. Default value is NULL.

   All of the configurations that are provided in QUERY_TIME_CONFIGURATIONS must be present in pg_settings. The configurations are set locally in the scope of the function. If you don't set any configurations, AlloyDB uses all of the configurations that were set in the session. You can also set only those configurations that are best suited for your use case.

   To see the default config settings, run the SHOW CONFIG_NAME command or view the pg_settings by running the following command:

   SELECT * FROM pg_settings;
   

   For more information, see pg_settings.

   If you have a ScaNN index for which you want to tune the recall, see the tuning parameters in ScaNN index reference.

   To feed the query input to the function, enclose the query in $$ on both sides.

   The following is an example output, where ann_execution_time is the time that it takes a vector query to execute using index scans. ground_truth_execution_time is the time that it takes the query to run using a sequential scan. You can use any PostgreSQL configuration that supports your use case.

   SELECT *
   FROM evaluate_query_recall(
   $$ SELECT id FROM t ORDER BY val <=> '[1,2,1]' LIMIT 10 $$, '{"scann.num_leaves_to_search":55}');
   id |                         query                          |          configurations           | recall | ann_execution_time | ground_truth_execution_time | index_type
   ----+--------------------------------------------------------+-----------------------------------+--------+--------------------+-----------------------------+------------
    1 |  SELECT id FROM t ORDER BY val <=> '[1,2,1]' LIMIT 10  | {"scann.num_leaves_to_search":55} |    0.2 |             16.772 |                        28.8 | scann
   (1 row)
   

Limitations

If a vector column that's used in a recall calculation query has a vector index other than ScaNN, then the evaluated recall might not be accurate.

What's next

• Create a ScaNN index.
• Store, index, and query vectors.
• Tune vector query performance.