Dimensionality reduction overview

Dimensionality reduction is the common term for a set of mathematical techniques used to capture the shape and relationships of data in a high-dimensional space and translate this information into a low-dimensional space.

Reducing dimensionality is important when you are working with large datasets that can contain thousands of features. In such a large data space, the wider range of distances between data points can make model output harder to interpret. For example, it makes it difficult to understand which data points are more closely situated and therefore represent more similar data. Dimensionality reduction helps you reduce the number of features while retaining the most important characteristics of the dataset. Reducing the number of features also helps reduce the training time of any models that use the data as input.

BigQuery ML offers the following models for dimensionality reduction:

• Principal component analysis (PCA)
• Autoencoder

You can use PCA and autoencoder models with the ML.PREDICT or ML.GENERATE_EMBEDDING functions to embed data into a lower-dimensional space, and with the ML.DETECT_ANOMALIES function to perform anomaly detection.

You can use the output from dimensionality reduction models for tasks such as the following:

• Similarity search: Find data points that are similar to each other based on their embeddings. This is great for finding related products, recommending similar content, or identifying duplicate or anomalous items.
• Clustering: Use embeddings as input features for k-means models in order to group data points together based on their similarities. This can help you discover hidden patterns and insights in your data.
• Machine learning: Use embeddings as input features for classification or regression models.