Stay organized with collections
Save and categorize content based on your preferences.
This page describes how you can use A/B experiments to understand how
Vertex AI Search for commerce is impacting your business.
Overview
An A/B experiment is a randomized
experiment with two groups: an experimental group and a control group. The
experimental group receives some different treatment (in this case, predictions
or search results from Vertex AI Search for commerce); the control group does not.
When you run an A/B experiment, you include the
information about which group a user was in when you record user events.
That information is used to refine the model and provide
metrics.
Both versions of your application must be the same, except that users in the
experimental group see results generated by Vertex AI Search for commerce and
the control group does not. You log user events for both groups.
For more on traffic splitting, see Splitting Traffic in the App Engine documentation.
Experiment platforms
Set up the experiment using a third-party experiment platform such as
VWO, AB Tasty. The control and experimental groups
each get a unique experiment ID from the platform. When you record a user event,
specify which group the user is in by including the experiment ID in the
experimentIds field. Providing the experiment ID lets
you to compare the metrics for the versions of your
application seen by the control and experimental groups.
Best practices for A/B experiments
The goal of an A/B experiment is to accurately determine the impact of updating
your site (in this case, employing Vertex AI Search for commerce). To get an accurate measure
of the impact, you must design and implement the experiment correctly, so that
other differences don't creep in and impact the experiment results.
To design a meaningful A/B experiment, use the following tips:
Before setting up your A/B experiment, use prediction or search preview to
ensure that your model is behaving as you expect.
Make sure that the behavior of your site is identical for the experimental
group and the control group.
Site behavior includes latency, display format, text format, page layout,
image quality, and image size. There should be no discernible differences
for any of these attributes between the experience of the control and
experiment groups.
Accept and display results as they are returned from
Vertex AI Search for commerce, and display them in the same order as they are
returned.
Filtering out items that are out of stock is acceptable. However, you should
avoid filtering or ordering results based on your business
rules.
If you are using search user events and include the required attribution token with them, make sure they are
set up correctly. See the documentation for Attribution tokens.
Make sure that the serving config you provide when you request
recommendations or search results matches your intention for that
recommendation or search result, and the location where you display the
results.
When you use recommendations, the serving config
affects how models are trained and therefore what products are recommended.
Learn more.
If you are comparing an existing solution with Vertex AI Search for commerce,
keep the experience of the control group strictly segregated from the
experience of the experimental group.
If the control solution does not provide a recommendation or search result,
don't provide one from Vertex AI Search for commerce in the control pages.
Doing so will skew your test results.
Make sure your users don't switch between the control group and the
experiment group. This is especially important within the same session, but
also recommended across sessions. This improves experiment performance and
helps you get statistically significant A/B test results sooner.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-28 UTC."],[],[],null,["# General guidance on conducting A/B experiments\n\nThis page describes how you can use A/B experiments to understand how\nVertex AI Search for commerce is impacting your business.\n\nOverview\n--------\n\nAn [A/B experiment](https://en.wikipedia.org/wiki/A/B_testing) is a randomized\nexperiment with two groups: an experimental group and a control group. The\nexperimental group receives some different treatment (in this case, predictions\nor search results from Vertex AI Search for commerce); the control group does not.\n\nWhen you run an A/B experiment, you include the\ninformation about which group a user was in when you record user events.\nThat information is used to refine the model and provide\nmetrics.\n\nBoth versions of your application must be the same, except that users in the\nexperimental group see results generated by Vertex AI Search for commerce and\nthe control group does not. You log user events for both groups.\n\nFor more on traffic splitting, see [Splitting Traffic](/appengine/docs/standard/python3/splitting-traffic) in the App Engine documentation. \n\nExperiment platforms\n--------------------\n\nSet up the experiment using a third-party experiment platform such as\n[VWO](https://vwo.com/), [AB Tasty](https://www.abtasty.com/). The control and experimental groups\neach get a unique experiment ID from the platform. When you record a user event,\nspecify which group the user is in by including the experiment ID in the\n`experimentIds` field. Providing the experiment ID lets\nyou to compare the metrics for the versions of your\napplication seen by the control and experimental groups.\n\nBest practices for A/B experiments\n----------------------------------\n\nThe goal of an A/B experiment is to accurately determine the impact of updating\nyour site (in this case, employing Vertex AI Search for commerce). To get an accurate measure\nof the impact, you must design and implement the experiment correctly, so that\nother differences don't creep in and impact the experiment results.\n\nTo design a meaningful A/B experiment, use the following tips:\n\n- Before setting up your A/B experiment, use prediction or search preview to\n ensure that your model is behaving as you expect.\n\n- Make sure that the behavior of your site is identical for the experimental\n group and the control group.\n\n Site behavior includes latency, display format, text format, page layout,\n image quality, and image size. There should be no discernible differences\n for any of these attributes between the experience of the control and\n experiment groups.\n- Accept and display results as they are returned from\n Vertex AI Search for commerce, and display them in the same order as they are\n returned.\n\n Filtering out items that are out of stock is acceptable. However, you should\n avoid filtering or ordering results based on your business\n rules.\n- If you are using search user events and include the required attribution token with them, make sure they are\n set up correctly. See the documentation for [Attribution tokens](/retail/docs/attribution-tokens).\n\n- Make sure that the serving config you provide when you request\n recommendations or search results matches your intention for that\n recommendation or search result, and the location where you display the\n results.\n\n When you use recommendations, the serving config\n affects how models are trained and therefore what products are recommended.\n [Learn more](/retail/docs/models).\n- If you are comparing an existing solution with Vertex AI Search for commerce,\n keep the experience of the control group strictly segregated from the\n experience of the experimental group.\n\n If the control solution does not provide a recommendation or search result,\n don't provide one from Vertex AI Search for commerce in the control pages.\n Doing so will skew your test results.\n\n Make sure your users don't switch between the control group and the\n experiment group. This is especially important within the same session, but\n also recommended across sessions. This improves experiment performance and\n helps you get statistically significant A/B test results sooner."]]
